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Abstract 

The  purpose  of  this  study  was  to  evaluate  the  Automated 
Cost  Estimating  Integrated  Tools  (ACEIT)  System  and  answer 
the  following  investigative  questions:  (1)  How  user-friendly 
is  ACEIT  and  does  it  require  extensive  training?  (2)  Does 
the  proper  use  of  ACEIT  require  cost  estimating  expertise  or 
does  the  system  prompt  even  a  novice  to  generate  an  accurate 
estimate?  (3)  Does  the  ACEIT  Statistics  Package  give  the 
same  estimates  and  statistics  as  the  Statistical  Analysis 
System?  (4)  What  are  some  of  the  potential  benefits  and 
problems  of  successfully  implementing  ACEIT  throughout  the 
Air  Force?  (5)  Can  ACEIT  be  quickly  and  effectively 
introduced  to  AFIT  Graduate  Cost  Analysis  students  who  are 
unfamiliar  with  the  system? 

Review  of  the  existing  literature,  discussions  with 
ACEIT  users,  personal  experience  with  the  system,  and 
comments  of  AFIT  students  were  used  in  answering  the  first 
two  questions.  The  determination  was  that  ACEIT  is  very 
user-friendly  and  does  not  require  extensive  training. 
However,  even  though  the  system  provides  excellent  prompts, 
it  is  designed  as  a  decision  support  system  not  an  expert 
system.  Thus,  some  cost  estimating  expertise  is  required  to 
generate  accurate  cost  estimates. 

Sample  data  sets  run  through  the  ACEIT  Statistical 
Package  and  the  Statistical  Analysis  System  produced 
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comparable  estimates  and  statistics  for  learning  curves, 
linear  and  log-linear  regressions. 

The  ACEIT  system's  major  benefit  may  also  be  its  major 
potential  pitfall.  Since  ACEIT  is  a  framework  to  use  in 
building  an  estimate,  organizations  can  fill  its  databases 
with  their  own  unique  work  breakdown  structures,  data,  and 
cost  estimating  relationships.  This  benefit  can  turn  into  a 
pitfall  if  the  databases  are  not  accurate.  Each 
organization  must  ensure  quality  control  of  its  own  unique 
ACEIT  databases. 

Results  of  a  brief  introduction  of  ACEIT  to  AFIT 
Graduate  Cost  Analysis  students  indicate  the  system  can  be 
quickly  and  effectively  introduced.  The  four  one-and-a- 
quarter-hour  periods  included  a  briefing,  a  hands-on 
demonstration  of  the  ACEIT  tutorial,  and  a  hands-on 
exercise . 

Recommendations  of  this  study  include  suggested  updates 
to  the  already  excellent  User's  Guide,  suggestions  for 
additional  ACEIT  capabilities,  presenting  the  ACEIT 
introduction  earlier  in  the  Graduate  program  and  expanding 
it  to  five  one-and-a-quarter-hour  periods.  Further  research 
should  examine  the  technicalities  of  creating  the 
organizational  databases  and  replication  of  an  estimate  from 
scratch. 
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AN  EVALUATION  OF  THE 

AUTOMATED  COST  ESTIMATING  INTEGRATED  TOOLS  (ACEIT)  SYSTEM 

I.  Introduction 

General  Issue 

This  thesis  evaluates  the  Automated  Cost  Estimating 
Integrated  Tools  (ACEIT)  System.  Tecolote  Research,  Inc, 
developed  ACEIT  for  Electronic  Systems  Division  (ESD)  under 
contracts  F19628-84-D-0019  and  F19628-88-D-0007 .  It  is 
designed  as  a  user-friendly  integrated  set  of  tools  to  allow 
one  to  define  the  program  to  be  estimated  and  build  and 
document  the  cost  estimate.  Although  only  received  by  the 
Air  Force  in  the  spring  of  1988,  ACEIT  generated 
considerable  interest  at  the  April  1988  Air  Force  Systems 
Command  (AFSC)  Cost  Symposium  (4:2).  The  following  agencies 
have  installed  or  are  in  the  midst  of  installing  ACEIT  (5): 
Electronic  Systems  Division  MA 
Air  Force  Cost  Center  Washington  DC 
Arnold  Engineering  Development  Center  TN 
Munitions  Systems  Division  FL 
Space  Systems  Division  CA 
Ballistic  Systems  Division  CA 

Strategic  Defense  Initiative  National  Test  Facility  CO 
Air  Force  Institute  of  Technology  (AFIT)  OH 
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Originally  designed  to  support  ESD,  ACEIT  currently 
runs  on  a  mainframe  (VAX)  computer  (although  a  personal 
computer  version  for  certain  portions  of  the  system  is  under 
development) ,  It  has  an  extensive  database  of  ESD  program- 
related  cost  factors;  document,  source,  and  cost  estimating 
relationship  (CER)  equation  libraries;  cost  models;  and 
cost,  schedule,  and  technical  information  (13: Sec  1,1). 

ACEIT  gives  the  user  the  "capability  to  search  for  and 
extract  data  from  the  databases  and  to  develop  analogies  and 
CERs  from  the  data"  (13:Sec  1,1).  Prompted  by  the  growing 
interest  in  ACEIT  throughout  the  Air  Force,  this  research 
provides  an  independent  review  of  ACEIT  and  an  examination 
of  the  issues  involved  in  expanding  ACEIT 's  use  beyond  ESD. 

Specific  Problem 

Since  ACEIT  is  so  new,  there  has  been  little  time  to  1) 
identify  errors  in  its  instructions  which  would  lead  a  user 
to  malce  an  erroneous  estimate  or  2)  identify  problems  which 
could  occur  as  ACEIT  is  implemented  Air  Force-wide. 

Research  Objective 

The  objective  of  this  research  is  to  provide  an 
independent,  in-depth  evaluation  of  ACEIT  by  developing 
familiarity  with  the  ACEIT  system,  sampling  the  system's 
da*: a,  interviewing  current  and  prospective  users,  and 
developing  an  "Introduction  to  ACEIT"  lesson  plan  for  Air 
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Force  Institute  of  Technology  (AFIT)  Graduate  Cost  Analysis 
students . 


Research  Investigative  Questions 

A  thorough  evaluation  of  ACEIT  requires  answers  to  the 
following  questions. 

1.  How  user-friendly  is  ACEIT?  Does  it  require 
extensive  training? 

2.  Does  proper  use  of  ACEIT  require  cost  estimating 
expertise  or  does  it  prompt  even  a  novice  user  to 
generate  accurate  estimates? 

3.  Does  the  ACEIT  Statistics  Package  give  the  same 
estimates  and  statistics  as  the  Statistical  Analysis 
System  (SAS)? 

4.  What  are  some  of  the  potential  benefits  and 
problems  to  be  avoided  in  successfully  implementing 
ACEIT  in  organizations  throughout  the  Air  Force? 

5.  Can  ACEIT  be  quickly  and  effectively  introduced  to 
the  AFIT  Graduate  Cost  Analysis  students  who  are 
unfamiliar  with  the  system? 


Assumptions 

Since  the  ACEIT  system  includes  a  built-in  tutorial 
program,  this  research  hypothesizes  that  the  tutorial  and 
documentation  will  provide  adequate  training  to  effectively 
use  the  system.  ESD  currently  hosts  a  5-day  training  course 
on  ACEIT.  However,  in  real  world  situations  individuals 
must  frequently  learn  to  use  systems  without  attending  a 
training  course.  Therefore,  this  research  should  test  the 
hypothesis,  provide  some  indication  of  how  adequate  the 
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existing  tutorial  is,  and  determine  whether  or  not  a 
training  course  should  be  considered  mandatory. 

Definitions 

Before  discussing  the  details  of  the  ACEIT  system  and 
addressing  the  research  investigative  questions,  it  is 
necessary  to  clear  up  the  confusion  indicated  in  various 
interviews  as  to  exactly  what  ACEIT  is  and  what  it  is  not. 
ACEIT  is  intended  to  be  an  effective  decision  support 
system;  it  is  not  a  knowledge  based  expert  system  nor  is  it 
just  a  computerized  spreadsheet.  One  concern  uncovered 
during  this  research  was  that  ACEIT  is  a  dangerous  concept 
because  it  is  an  automated  cost  estimator  (or  expert  system) 
which  takes  the  individual  out  of  the  cost  estimating 
process.  At  the  other  extreme,  another  concern  was  that 
ACEIT  doesn't  give  the  user  anything  more  than  a  spreadsheet 
program  does.  Both  of  these  concerns  are  unwarranted. 

ACEIT  will  not  make  a  decision  for  a  user,  and  most 
certainly  it  does  not  take  the  individual  out  of  the 
estimating  process.  However,  ACEIT  will  provide 
significantly  more  support  to  the  estimator  than  a  simple 
spreadsheet  program. 

Davis  and  Olson  in  Management  Information  Systems 

define  decision  support  systems  as 

a  class  of  systems  which  support  the  process  of  making 
decisions.  The  emphasis  is  on  "support"  rather  than  on 
automation  of  decisions.  Decision  support  systems 
allow  the  decision  maker  to  retrieve  data  and  test 
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alternative  solutions  during  the  process  of  problem 
solving.  (6:368) 

They  further  define  the  concept  of  decision  support 
systems  in  terms  of  the  role  the  computer  plays  in  an 
effective  decision  making  process: 

1.  The  computer  must  support  the  manager  but  not 
replace  his  or  her  judgement.  It  should  therefore 
neither  try  to  provide  the  "answers"  nor  impose  a 
predefined  sequence  of  analysis. 

2.  The  main  payoff  of  computer  support  is  for 
semistructured  problems,  where  parts  of  the  analysis 
can  be  systematized  for  the  computer,  but  where  the 
decision  maker's  insight  and  judgement  are  needed  to 
control  the  process. 

3.  Effective  problem  solving  is  interactive  and  is 
enhanced  by  a  dialogue  between  the  user  and  the  system. 
The  user  explores  the  problem  situation  using  the 
analytic  and  information-providing  capabilities  of  the 
system  as  well  as  human  experience  and  insights. 
(6:368-369) 

The  key  comments  to  be  emphasized  in  the  above  passage 
clearly  address  the  concern  that  ACEIT  will  automatically 
make  the  cost  estimate  without  human  intervention.  ACEIT 
does  not  "replace"  the  manager's  judgement.  ACEIT  does  not 
impose  a  predefined  sequence  of  analysis  (although  it  does 
provide  a  sequence  should  the  user  choose  to  follow  it) . 
ACEIT  does  systematize  the  calculations  required  in  a  cost 
analysis  (such  as  applying  inflation  rates)  but  the  user's 
"insight  and  judgment  are  needed  to  control  the  process" 
(6:368).  ACEIT  is  interactive. 

ACEIT  is  not  an  expert  system.  According  to  Davis  and 
Olson, 

An  expert  system  is  a  computer  application  that  guides 
the  performance  of  ill-structured  tasks  which  usually 


5 


require  experience  and  specialized  knowledge  . . .  Using 
an  expert  system,  a  non-expert  can  achieve  performance 
comparable  to  an  expert  in  that  particular  problem 
domain.  ...  The  unique,  distinguishing  feature  of  an 
expert  system  is  the  knowledge  base,  the  data  and 
decision  rules  which  represent  the  expertise.  (6:375) 

An  example  of  a  rule-based  expert  system  would  be  the 

medical  diagnosis  system  MYCIN  (20:93).  With  this  type  of 

system  the  user  provides  a  set  of  symptoms  and  the  system 

then  searches  its  knowledge  base  of  symptoms  and  possible 

causes  (6:375).  ACEIT  is  not  an  expert  system;  it  has  no 

knowledge  base  or  decision  rules. 

ACEIT  clearly  fits  in  the  decision  support  system 

category.  As  currently  configured,  ACEIT  consists  of  eight 

integrated  modules: 

Automated  Cost  Estimator  (ACE) 

Automated  Cost  Database  (ACDB) 

Automated  Cost  Document  Library  (ACDL) 

Cost  Estimating  Relationship  Library  (CERL) 

Cost  Estimating  Models  (CEM) 

Software  Cost  Database  (SCDB) 

Software  Size  Database  (SSDB) 

Statistics  Package  (STATPAK) 

ACE  is  the  "heart  of  ACEIT"  being  the  module  from  which  all 
the  other  modules  can  be  accessed  (13: Sec  1,1).  A 
discussion  of  each  module  will  be  presented  in  Chapter  II. 
Suffice  it  to  say  at  this  point  that  the  modules  together 
clearly  meet  the  definition  of  a  decision  support  system. 


6 


Another  misconception  about  ACEIT  is  that  it  is  a 
model.  As  is  evident  from  the  list  of  ACEIT  modules  above, 
one  module  within  ACEIT  provides  access  to  a  variety  of 
models.  However,  ACEIT  itself  is  not  a  model.  The 
predictive  accuracy  of  an  estimate  developed  using  ACEIT 
does  not  reflect  on  ACEIT 's  ability  to  predict.  ACEIT  does 
not  predict  cost,  rather  the  methodology  input  into  the 
ACEIT  framework  is  predicting  the  cost.  It  is  that 
methodology,  not  ACEIT,  which  would  then  be  either  accurate 
or  inaccurate.  In  the  words  of  Ms  Ellen  Coakley,  ESD/ACC, 
Technical  Director,  "ACEIT  is  an  architecture"  (5) .  ACEIT 
is  a  framework  to  use  in  building  a  cost  estimate.  It 
provides  access  to  a  lot  of  data,  CERs,  and  models,  as  well 
as  doing  learning  curve  and  inflation  calculations,  but  it 
is  not  a  model.  Therefore,  there  is  really  no  need  to 
validate  ACEIT  as  an  accurate  cost  estimating  model; 
instead,  all  of  the  data,  CERs,  and  models  put  into  any 
given  ACEIT  database  need  to  be  validated.  This  thesis  will 
evaluate  the  STATPAK  and  learning  curve  calculation  routines 
within  ACEIT.  However,  it  must  be  made  clear  that  saying  an 
estimate  was  built  using  ACEIT  is  not  like  saying  an 
estimate  was  built  using  RCA  PRICE-S  or  COCOMO  for  instance. 
ACEIT  is  simply  a  framework  which  provides  access  to  the 
tools  necessary  to  make  an  estimate. 
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Scope 

The  size  and  complexity  of  the  ACEIT  system  necessitate 
that  the  scope  of  this  research  effort  be  limited.  Part  of 
this  research  includes  evaluating  the  STATPAK  and  learning 
curve  calculations.  Since  it  would  be  impossible  to  test 
every  conceivable  type  of  data  or  equation  in  the  short  time 
allotted  for  this  research,  representative  data  will  be 
used.  Additionally,  the  inflation  calculations  will  not  be 
evaluated  at  all.  The  current  version  of  ACEIT  does  not 
permit  a  user  access  to  the  inflation  indices  used  in  the 
actual  calculations.  They  are  referenced  internally  and  the 
calculations  are  transparent  to  the  user.  Therefore,  it 
would  be  somewhat  difficult  to  verify  the  exact  numbers 
being  used.  Another  consideration  (described  in  Chapter  IV) 
is  the  fact  that  Bradley's  replication  of  an  existing  cost 
estimate  verified  that,  in  that  case,  the  inflation 
calculations  were  correct  (4:19).  Thus,  rechecking  the 
inflation  calculations  is  not  a  priority  concern  and  will 
not  be  included  in  this  research. 
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II.  Literature  Review 


Introduction 

Since  ACEIT  is  so  new,  there  are  not  any  published 
reviews  of  it.  The  only  published  reference  to  ACEIT  is  the 
User's  Guide  which  is  used  extensively  for  this  research. 
ACEIT  does  not  show  up  in  any  computerized  or  manual 
literature  search  under  cost  model,  automated  cost  model, 
cost  effectiveness,  cost  estimating,  or  automated  cost 
estimating.  Therefore,  this  chapter  first  provides  a 
thorough  description  of  each  of  the  ACEIT  system  modules  to 
help  the  reader  unfamiliar  with  ACEIT  to  understand  the 
remainder  of  the  thesis.  Next,  literature  on  systems  which 
may  compare  to  ACEIT  is  reviewed.  This  is  followed  by  an 
examination  of  several  unpublished  reviews  of  ACEIT. 

Lastly,  literature  pertaining  to  the  concepts  of  user- 
friendliness,  cost  estimating  expertise,  and  statistics 
package  accuracy  is  reviewed. 

ACEIT  System  Description 

Automated  Cost  Estimator  (ACE) . 

ACE  is  the  "heart"  of  ACEIT,  since  it  is  the  module 
from  which  every  other  module  can  be  accessed  (13: Sec  1,1). 
"Ace  uses  a  user-friendly  spreadsheet  architecture  employing 
on-line  help  and  'Lotus'  style  command  menus"  (13:Sec  1,1). 
ACE  currently  has  two  operating  modes,  one  for  the  novice 
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estimator  and  the  other  for  the  experienced  estimator.  The 
difference  in  the  two  modes  is  the  level  of  prompts 
automatically  provided.  "The  expert  mode  provides  nominal 
guidance  and  assumes  that  you  are  an  experienced  cost 
estimator,  familiar  with  the  ACE  system  software  operation" 
(4:7).  The  novice  mode  includes  a  set  of  about  100  tutorial 
training  screens  created  by  Ms  Ellen  Coakley  (ESD/ACC) . 

These  screens  essentially  walk  a  novice  estimator  through 
the  estimating  process  using  ACEIT  with  "tutorials,  on¬ 
screen  prompts,  and  interactive  sample  problems"  (13: Sec 
1,1).  Even  if  a  user  is  in  the  expert  mode,  he  can  access 
the  novice  mode  screens  if  desired.  These  screens  cover  all 
the  essentials:  defining  the  estimating  task;  building  and 
defining  a  work  breakdown  structure  (WBS) ;  planning  the  cost 
estimating  methodology  for  each  WBS  element;  including 
General  and  Administrative  overhead  and  Fee  rate 
adjustments;  addressing  learning  curve  methodology; 
collecting,  evaluating,  and  normalizing  data;  addressing 
risk  and  potential  contract  changes;  calculating  the 
estimate  in  base  year  dollars;  time  phasing  the  estimate; 
escalating  the  estimate  to  then-year  dollars;  and  finally 
documenting  the  estimate  (4:36-110).  The  screens  do  not 
make  the  novice  mode  an  expert  system  even  though  Ms  Coakley 
is  an  expert  and  they  contain  her  advice.  The  screens  are 
instead  a  computerized  version  of  a  cost  estimating 
handbook.  They  could  just  as  well  have  been  located  in  an 
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off-line  workbook,  but  computerizing  the  text  makes  the 
learning  an  interactive  process  versus  a  passive  reading 
process . 

Whichever  mode  is  used,  ACE  will  allow  a  user  to  create 
an  estimate.  ACE  includes  built-in  WBSs  consistent  with 
Military  Standard  881A  (13:Sec  2,21);  these  include  separate 
WBSs  for  aircraft,  electronics,  missile,  ordnance,  ship, 
space,  and  surface  vehicle  systems  (4:42).  However,  the 
user  has  the  capability  to  tailor  any  system  WBS  to  his 
specific  requirements.  ACE's  system  WBS  elements  are  linked 
to  ACE  built-in  libraries  of  CERS ,  models,  and  sources 
(people,  places,  documents,  and  databases)  of  data/knowledge 
(13:Sec  1,2).  The  link  allows  ACE  to  present  the  estimator 
with  only  those  CERs,  models,  and  sources  applicable  to  that 
particular  WBS  element  instead  of  the  entire  library 
(although  the  entire  library  can  still  be  accessed) . 
Additionally,  the  linkage  will  carry  information  on  fee,  G&A 
rate,  learning  curve  etc.  throughout  the  estimate.  The  CER 
library  within  ACE  is  not  the  Cost  Estimating  Relationship 
Library  (CERL)  which  will  be  described  later.  The  ACE  CERs 
are  a  separate  library  of  CERs.  ESD  has  chosen  to  include 
only  those  CERs  which  have  been  "identified/developed  and 
reviewed  for  ESD  use  by  ESD/ACC"  in  the  ACE  built-in  library 
(4:63).  The  CERs  in  the  built-in  library  also  include 
comments  on  the  parameters,  definitions,  data  ranges, 
statistics,  and  recommended  uses  (13:Sec  1,2).  Any  selected 
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CERs  can  be  electronically  transferred  to  the  ACE  workscreen 
to  be  used  in  building  an  estimate.  ACE  also  allows  a  user 
to  run  a  model  from  ACE's  built-in  library  and  transfer  the 
answer  to  the  worksheet  (13:Sec  1,2).  Additionally,  ACE 
lets  a  user  search  on-line  databases  to  find  an  analogy,  or 
transfer  data  to  the  STATPAK  module  to  create  a  new  CER 
(13:Sec  1,2) . 

Before  calculating  an  estimate,  ACE  checks  the 
methodology  for  missing  appropriations,  equations,  parameter 
input  values,  and  proper  format  as  well  as  prompting  the 
user  for  learning  curve  information  (13:Sec  1,2).  Following 
the  completion  of  the  estimating  methodology,  ACE  moves  the 
user  to  a  calculation  workscreen  where  inputs  can  be  varied 
to  run  multiple  simultaneous  sensitivity/what-if  analyses 
(13:Sec  1,2).  ACE  uses  built-in  learning,  inflation,  and 
summing  routines  to  translate  figures  into  the  units  and 
base  year  specified  by  the  user  {13:Sec  1,2).  ACE  also  time 
phases  the  estimate  and  then  translates  it  into  then-year 
dollars  based  on  the  user's  direction  (13:Sec  1,3). 

Finally,  ACE  will  print  out  any  of  the  following 
documentation  requested  by  the  user: 

1.  Summary  Costs  (Base-Year  and  Then-Year)  by 

Appropriation  by  Fiscal  Year 

2.  Time  Phased  Dollars  (Base-Year  and  Then-Year) 

3.  Estimates  in  Base-Year  Dollars 

4.  Estimating  Methodology 

5.  Time  Phasing  Methodology 
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6.  WBS  Definitions 


7.  CER/Source/Model  Definitions 

8.  ACE  Notepad  Comments 

A  key  aspect  of  the  ACE  module  which  makes  it  so 
flexible  is  the  fact  that  the  system  manager  has  a  set  of 
utilities  available  to  tailor  system  training  screens,  WBSs, 
definitions,  expansions,  and  built-in  libraries  (13: Sec 
1,3).  As  mentioned  earlier,  ACEIT  can  be  considered  a 
framework  for  building  an  estimate.  The  ability  to  tailor 
training  screens,  WBSs,  CER  libraries,  etc.  to  meet  an 
organization's  unique  requirements  is  what  will  make  it 
possible  for  units  Air  Force-wide  to  effectively  use  ACEIT. 
This  subject  will  be  addressed  further  in  Chapter  IV. 

Automated  Cost  Database  (ACDB) . 

The  Automated  Cost  Database  currently  contains  cost, 
schedule,  technical,  and  programmatic  data  on  over  130  ESD 
system  acquisition  contracts  {13:Sec  1,3).  Monthly  data  was 
extracted  from  CPRs ,  C/SSRs,  and  CFSRs .  Following  ESD/ACC's 
approval  of  a  mapping  scheme,  this  data  was  then  mapped  into 
a  standard  WBS  for  electronic  systems  (13: Sec  1,3).  This 
data  can  be  queried  to  find  an  analogous  program,  system 
contractor,  or  contract  (13:Sec  1,3).  Reports  by  WBS,  in 
then-year  or  base-year  dollars,  can  be  obtained  which 
include  any  of  the  following:  actual  or  cumulative  cost  by 
month,  percent  spent  by  month,  percent  spent  by  schedule 
milestones,  cost-to-cost  factors,  durations,  lag  times,  and 


13 


cumulative  beta  curve  parameters  (13: Sec  1,3).  An  important 
feature  of  the  ACDB  which  makes  a  cost  estimator's  work  a 
lot  easier  is  that  data  selected  from  the  ACDB  can  be 
electronically  transferred  to  the  STATPAK  module  for 
analysis.  This  is  a  fast,  error-free  (no  typos)  way  to 
create  CERs  from  analogous  programs.  However,  as  in  any 
estimate  whether  computerized  or  not,  the  estimator  must 
make  sure  the  programs  used  are  truly  analogous.  ACEIT  will 
not  do  this  for  a  user,  it  only  provides  easy  access  to  the 
data  from  the  computer  terminal. 

Automated  Cost  Document  Library  (ACDL) . 

The  Automated  Cost  Document  Library  contains  abstracts 
on  over  1200  ESD  technical  and  cost  related  source  documents 
{13:Sec  1,4).  The  ACDL  can  be  searched  by  Title,  Author, 
Subject,  Abstract  Text,  Publication  Date,  and/or  Originating 
Organization  (13:Sec  1,4).  The  ACDL  is  simply  a 
computerized  card  catalog  of  the  ESD  Cost  Library,  The 
benefit  of  having  the  ACDL  module  is  that  it  simplifies  the 
estimator's  search  for  documents  related  to  his  program. 
Although  many  libraries  now  have  computerized  card  catalogs 
(Wright  State  University  in  Dayton,  Ohio  for  instance) ,  it 
is  a  nice  feature  for  the  cost  estimator  to  be  able  to 
quickly  search  from  his  terminal  while  doing  an  estimate, 
ACDL  is  not  a  substitute  for  reading  a  document,  it  just 
points  the  user  in  the  right  direction. 
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Cost  Estimating  Relationship  Library  (CERL) 

The  Cost  Estimating  Relationship  Library  contains  over 
1150  CERs  extracted  from  more  than  380  cost  studies 
performed  for  the  Army,  Air  Force,  and  Navy  (13: Sec  1,4). 

The  library  is  adapted  from  the  MICOM  Automated  Database 
System  (MIDAS)  developed  by  the  Huntsville,  Alabama  office 
of  Tecolote  Research,  Inc.  for  the  U.S.  Army  Missile  Command 
Comptroller  (13:Sec  5,i).  In  other  words,  the  library  was 
already  available  and  only  had  to  modified  to  be  included  in 
ACEIT.  An  important  point  to  mention  here  is  that  the  CERs 
contained  in  the  library  have  not  been  evaluated  by  ESD, 
they  have  only  been  compiled.  Included  in  the  library 
extracts  are  summary  documentation  for  each  CER  such  as 
source  data  and  relevant  descriptive  statistics  (13: Sec 
1,4).  However,  an  estimator  would  be  wise  to  consult  the 
original  source  document  on  any  CER  under  consideration. 

Cost  Estimating  Models  (CEM) .  (13:Sec  1,4-5) 

Cost  Estimating  Models  is  a  large  module  consisting  of 
a  set  of  Tecolote-developed  and  commercially  available  cost 
estimating  models  including  BBEST,  COCOMO,  WICOMO,  RADARDEV, 
and  access  to  the  RCA  PRICE  Family.  BBEST  is  a  family  of 
twelve  electronics  hardware  cost  models.  COCOMO  and  WICOMO 
estimate  software  cost  and  schedule.  RADARDEV  is  an 
automated  set  of  CERs  for  estimating  the  complete 
development  costs  of  radar/C^  equipment  at  the  system  or 


subsystem  level.  The  RCA  PRICE  family  of  models  are 


applicable  for  a  variety  of  electronic  hardware,  circuit 
cards,  software,  etc.  Although  the  PRICE  models  can 
interface  with  ACEIT,  this  feature  was  not  available  on  the 
ACEIT  system  as  it  was  configured  at  AFIT. 

Software  Cost  Database  (SCDB).  (13:Sec  1,6) 

The  Software  Cost  Database  currently  is  not  integrated 
into  ACEIT  (a  future  version  will  be)  but  instead  is 
installed  as  a  stand  alone  system  on  the  ESD  base  VAX. 
Similar  to  the  Automated  Cost  Database  and  the  Software  Size 
Database,  SCDB  contains  detailed  cost,  schedule,  and 
technical  data  for  28  software  development  projects, 
containing  177  computer  software  configuration  items 
(CSCIs)  . 

Software  Size  Database  (SSDB)  .  (13:Sec  1,6) 

The  Software  Size  Database  contains  software  size, 
functions,  and  technical  information  for  1,458  computer 
software  configuration  items.  Since  the  SSDB  is  integrated 
into  ACEIT,  the  user  can  search  and  retrieve  data  for 
reports  or  to  be  moved  into  STATPAK  for  analysis. 

Statistics  Package  (STATPAK). 

STATPAK  is  a  statistical  package  for  analysis  of  data 
extracted  from  ACEIT  and  transferred  into  STATPAK  or  data 
from  other  sources  (13:Sec  1,6).  The  ACEIT  User's  Guide 
states 

The  main  purpose  of  this  software  is  to  run  regression 
analysis  for  both  linear  (or  log-linear)  and  nonlinear 
models  in  a  user-friendly  fashion.  Learning  curve 
(unit  and  cumulative  average)  theories  are  also 
incorporated.  Another  option  provides  "descriptive 
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statistics."  The  basic  descriptive  statistics  are  the 
measures  of  central  tendency  (location)  and  the 
measures  of  dispersion  (scatter)  of  the  data  set. 

These  statistics  are  used  to  describe,  graphically  and 
numerically,  the  characteristic  behaviors  of  sets  of 
data.  (13:App  A,l) 

STATPAK  allows  the  user  to  edit  data,  review  histograms  and 
scatterplots  and  perform  basic  statistical  analysis, 
regression,  and  learning  curves  (13:Sec  1,6). 

Other  Systems 

There  is  very  little  literature  identified  on  any 
systems  which  are  comparable  to  ACEIT.  Although  there  are 
numerous  automated  cost  estimating  models,  none  allow  a  user 
to  both  access  a  variety  of  models  and  actually  create  CERs . 
ACEIT  appears  to  be  a  truly  innov?>'-’  a  idea. 

A  good  example  of  ♦•he  automated  models  which  are 
available  is  the  Army  system  described  in  "Beyond  the  DCA-P- 
92(R)  Structure  Costing  at  a  New  Level  of  Detail"  (24). 

This  paper  described  "an  automated  cost  estimating  tool" 
which  uses  existing  models  to  generate  a  "baseline  cost 
estimate"  (24:1).  The  baseline  cost  estimate  "represents  an 
operational,  integrated  cost  estimating  process  for  both  the 
acquisition  and  the  Planning,  Programming,  Budgeting  and 
Execution  cycles"  (24:2).  However,  this  tool  does  not 
generate  CERs  and  it  does  not  provide  the  flexibility  that 
ACEIT  does. 

The  system  which  most  closely  resembles  ACEIT  is  the 
Air  Force  Cost  Center's  (AFCSTC)  Cost  Estimating  System 
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(CES)  developed  by  Delta  Research  Corporation.  In  the 
user's  guide  cover  letter,  LeRoy  Baseman,  Technical  Director 
AFCSTC,  states  CES  "can  be  used  by  analysts  to  develop  and 
test  cost  estimating  relationships"  (8) .  Unlike  the  current 
version  of  ACEIT,  CES  runs  on  the  Zenith  248  or  any  IBM- 
PC/XT/AT  compatible  microcomputer  with  640K  of  RAM.  The  CES 
uses  a  separate  off-line  Aircraft  Cost  Handbook  to 
supplement  the  databases  stored  on  diskette.  While  ACEIT 
provides  a  tutorial  especially  designed  for  the  novice 
estimator,  the  CES  appears  to  be  geared  toward  the  more 
experienced  analyst. 

Unpublished  ACEIT  Reviews 

Although  no  published  reviews  of  ACEIT  were  identified, 
several  unpublished  papers  were  obtained. 

The  Air  Force  Institute  of  Technology  (AFIT)  Graduate 
Cost  Analysis  Class  88S  prepared  a  "Preliminary  Evaluation 
of  the  Automated  Cost  Estimating  Integrated  Tools  (ACEIT) 
System."  Their  review  focused  on  the  "user-software 
integration  of  ACEIT  . . .  using  a  cost  estimating  case  study 
provided  by  Electronics  Systems  Division"  (1:1).  The  class 
determined  ACEIT  to  be  "a  very  powerful  package  with  great 
potential"  and  offered  some  minor  recommendations  not  to 
change  the  architecture  of  ACEIT  but  to  "make  the  system 
less  cumbersome"  and  "to  ease  working  within  the 
architecture"  (1:5).  Since  their  report  was  completed,  an 
updated  version  of  ACEIT  was  installed  on  the  AFIT  computer. 
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This,  plus  some  changes  made  by  the  AFIT  Database 
Administrator  and  a  reconfiguration  of  some  of  the  AFIT 
terminals,  significantly  improved  user-system  interaction. 
The  system  installed  at  AFIT  is  now  much  easier  to  access, 
and  updating  and  printing  of  files  is  also  very 
straightforward . 

Two  other  reports  were  prepared  by  Aeronautical  Systems 
Division  (ASD)  cost  personnel.  In  the  first,  Kenneth  L. 
Birkofer,  ASD/ACCR,  in  "A  Review  of  the  ESD  ACEIT  Model" 
briefly  described  the  features  of  the  eight  ACEIT  modules, 
their  ease  of  use,  and  their  potential  "usefulness  to  the 
ASD  cost  estimating  community"  (3:1).  Randall  S.  Bradley 
prepared  the  second  report  to  fulfill  the  research  project 
requirement  within  the  ASD/AC  Comptroller  Training  Program 
(4:3).  He  basically  evaluated  the  spreadsheet  computational 
features  and  ease  of  use  of  the  ACE  module  by  replicating 
the  ASD  cost  estimate  for  the  Light  Detecting  and  Ranging 
(LIDAR)  System.  He  did  not  evaluate  any  of  the  database, 

CER  library,  model  library,  learning  curve,  or  STATPAK 
capabilities.  A  further  analysis  of  both  of  these  reports 
is  included  in  Chapter  IV. 

User-Friendliness 

Literature  pertaining  to  the  concept  of  user- 
friendliness  was  reviewed  to  establish  guidelines  in 
evaluating  ACEIT' s  user-friendliness.  Unfortunately,  the 
review  has  indicated  that  user-friendly  is  hard  to  define. 
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Webster's  Dictionary  does  not  even  provide  a  definition. 

However,  Henry  Simpson  wrote  a  whole  book  for  small  computer 

programers  to  help  them  write  user-friendly  programs.  His 

comments  also  apply  to  large  mainframe  computer  programs 

like  ACEIT,  In  Design  of  User-Friendly  Programs  for  Small 

Computers .  Simpson  defines  a  user-friendly  program  as  "one 

with  features  that  acknowledge  human  factors"  and  one  which 

is  in  general  "easy  to  use,  tolerant  of  operator  errors, 

easy  to  learn,  and  acknowledges  that  human  beings  are 

imperfect  creatures"  (21:2).  Simpson  also  addresses  the 

importance  of  user  documentation,  stating  "It  is  accurate  to 

say  that  user  documentation  will  make  the  difference  between 

whether  your  program  can  be  used  effectively  or  not" 

(21:184).  Simpson  further  breaks  down  his  analysis  of  user 

documentation  into  "internal"  documentation  consisting  of 

comments  or  help  screens  that  can  be  accessed  from  within 

the  program  and  "external"  documentation  such  as  a  user's 

guide  or  video  tape  (21:193).  The  documentation  provided 

must  also  serve  as  both  a  tutorial  and  a  reference  guide: 

as  people  use  a  program,  they  gain  skill  and  their 
needs  change.  New  operators  need  a  step-by-step 
tutorial  that  they  can  use  to  work  through  the  program 
and  that  will  help  them  to  develop  skill  and 
confidence.  Experienced  operators  no  longer  need  this 
but  they  do  need  quick  access  to  important  reference 
information  for  using  the  program.  In  sum,  the  user's 
guide  should  serve  the  needs  of  all  members  of  its 
audience  and  should  accommodate  itself  to  growth  in 
skill.  (21:193) 

William  J.  Raduchel  approached  user-friendliness  from  a 
completely  different  perspective  than  Simpson.  In  a  Byte 


20 


magazine  article,  Raduchel  presented  a  mathematical  formula 
for  determining  user-friendliness  (19:103-104).  Before 
developing  the  formula,  he  made  several  comments  about  the 
concept  of  user-friendliness  in  general  such  as  "nobody  can 
define  what  user-friendly  means,  although  many  claim  to  know 
it  when  they  see  it”  and  "since  every  piece  of  software  is 
presumably  friendly  to  its  author,  every  piece  of  software 
can  be  described  as  user-friendly"  (19:101).  In  another 
comment  he  presents  a  simple  definition  of  user-friendly: 

"30  minutes  (or  less)  of  training  is  required  for  the 
software  to  be  usefully  applied"  (19:103). 

The  main  focus  of  Raduchel 's  article  is  his  formula 
based  on  the  premise  that  "no  system  can  be  user-friendly 
except  in  the  context  of  specific  problems  for  specific 
users"  (19:103).  Raduchel’s  logic  is  that  "A  system  is 
user-friendly  if  it  solves  problems  reliably"  and  that  the 
probability  of  solution,  F,  is  the  result  of 

-  the  probability  that  the  user  will  find  the  set  of 
steps  to  solve  the  problem 

p  -  the  probability  that  the  user  can  successfully 
execute  each  step 

n  -  the  minimum  possible  number  of  steps  in  the 
solution.  (19:103-104) 

Raduchel  uses  the  above  variables  to  generate  the  following 

equation  for  the  probability  of  solution: 

F  =  (P^)  *  (p")  (1) 


21 


His  requirement  for  being  user-friendly  is  "that  F  be  at  or 
above  some  threshold  probability  value  F^,  determined  by  the 
characteristics  of  the  alternative  systems"  (19:104). 

Raduchel  also  discusses  the  tradeoffs  implied  in  the 
equation.  For  instance,  if  the  number  of  steps  is  low  the 
probability  of  finding  the  correct  set  of  steps  will  be  high 
but  then  only  simple  problems  can  be  solved.  If  more 
complex  problems  are  to  be  solved  n  will  increase  and 
will  decrease.  The  result  is  "the  ease  of  use  and  ease  of 
learning  conflict".  However,  with  training  and  experience, 
the  user  "will  consider  a  system  with  fewer,  but  ...  more 
complex  steps  to  be  more  user-friendly".  Then,  as  also 
mentioned  by  Simpson,  to  be  user-friendly  the  system  must 
grow  with  the  user,  being  easy  to  learn  and  use  in  the 
beginning  yet  have  the  flexibility  to  skip  novice  steps  and 
solve  complex  problems  as  the  user  gains  expertise. 

(19:104) 

Cost  Estimating  Expertise 

A.D.  Kazanowski,  wrote  an  interesting  article  "Cost- 

Effectiveness  Fallacies  and  Misconceptions  Revisited,"  which 

tied  the  experience  of  a  cost  estimator  to  the  quality  of 

his  estimates.  He  comments  that  the  worth  of  a  cost- 

effectiveness  evaluation  is 

closely  correlated  with  the  experience  of  the  analyst. 
The  validity  of  many  evaluations  was  not  constrained  by 
the  limitations  of  cost-effectiveness  per  se  as  much  as 
by  the  ability  of  the  analyst  to  avoid  various  pitfalls 
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that  tended  to  invalidate  or  at  least  bias  the 
conclusions.  (12:151) 

He  presents  about  two  dozen  problem  areas  including  criteria 
selection,  assuming  all  pertinent  criteria  can  be 
quantified,  using  criteria  because  they  are  quantifiable  or 
ignoring  criteria  because  they  are  not  quantifiable,  and 
extrapolating  the  results  of  the  evaluation  beyond  the 
relevant  range  (12).  Since  the  ACEIT  system  is  specifically 
designed  for  both  novice  and  expert  estimators,  it  will  be 
interesting  to  discover  if  the  documentation  and  structure 
of  the  system  encourages  a  novice  to  either  make  or  avoid 
the  pitfalls  noted  above. 

Statistics  Package  Accuracy 

James  W.  Longley's  article,  "An  Appraisal  of  Least 
Squares  Programs  for  the  Electronic  Computer  From  the  Point 
of  View  of  the  User",  written  in  1967,  is  still  applicable 
today.  The  article  illustrates  that  the  procedures  used  by 
computers  as  they  carry  through  their  calculations  can 
result  in  very  large  errors  which  may  only  be  evident  with 
certain  data  (14).  Therefore,  data  which  challenges  the 
regression  program  will  highlight  programs  which  are  not 
effective  for  all  data  even  though  they  seem  to  be  producing 
reasonable  regression  solutions  for  some  data.  Since  the 
time  constraints  imposed  on  this  research  prevent  every 
conceivable  type  of  data  from  being  tested  on  the  ACEIT 
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STATPAK  module,  Longley's  data  should  provide  insight  as  to 

how  well  STATPAK  handles  a  challenging  data  set. 

Longley  ran  economic  data  (some  of  which  is  extracted 

in  Appendix  G)  on  a  series  of  available  programs  on  various 

computers  (14:827).  The  data  used  was  interesting: 

. . .  the  means  of  the  vectors  of  X  and  the  mean  of  the 
total  Y-variable  end  in  round  numbers,  but  also  the 
elements  in  the  product  moments  matrix  . . .  ended  in 
either  a  five  or  zero  within  nine  decimals.  ...  While 
the  determinant  of  the  6x6  product  moments  matrix  was 
5.7  septillions  (25  digits  to  the  left  of  the  decimal), 
the  determinant  of  the  correlation  matrix  was 
0.000  000  015  796  154  862,  which  proves  that  neither 
matrix  is  singular.  (14:820) 

If  the  matrix  was  singular,  its  inverse  would  not  exist  (due 
to  the  determinant  equalling  zero)  and  the  regression 
equation  could  not  be  computed.  However,  regression 
equations  could  be  computed  for  Longley's  data,  although  the 
very  small  determinant  helps  highlight  program  and  computer 
weaknesses.  The  fact  that  the  product  moments  matrix  ended 
in  either  a  five  or  zero  accentuated  the  differences 
produced  by  either  rounding  or  truncating  in  the 
calculations . 

Even  more  interesting  than  the  characteristics  of  the 

data,  were  the  results  obtained  from  running  that  data  on 

different  computers  with  different  programs: 

Many  programs  tested  with  identical  inputs  produced 
results  which  differed  from  each  other  in  every  digit 
for  all  multivariate  equations  run.  In  some  instances 
the  sign  of  the  net  regression  coefficients  were  wrong. 
(14:827) 
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Although  the  same  data  inputs  were  being  used,  the 
differences  in  how  the  computations  were  actually 
accomplished  (how  digits  were  carried  in  the  calculations, 
and  whether  rounding  or  truncating  was  used)  produced  a 
variety  of  resulting  regression  coefficients. 

Longley  compared  his  test  program  results  against 
"values  for  some  165  equations  carefully  worked  out  on  a 
desk  calculator"  which  agreed  with  an  available  matrix 
inversion  program  for  an  IBM  1401  computer  modified  to  carry 
matrix  calculations  to  40  digits  (14:821-822). 

It  is  the  intent  of  this  research  to  run  Longley 's  data 
set  through  the  ACEIT  STATPAK  program  and  compare  the 
results  to  Longley 's  calculated  values. 

Comments  on  the  Literature  Reviewed 

The  unpublished  reports  of  ACEIT  along  with  the 
definitions  of  a  decision  support  system  and  user- 
friendliness  provide  the  background  needed  to  evaluate  the 
ACEIT  system.  It  would  be  convenient  to  use  Raduchel ' s 
simple  "30  minutes  (or  less)  of  training"  definition  of 
user-friendliness.  However,  ACEIT  is  too  involved  to  be 
usefully  applied  after  only  30  minutes  of  training.  The 
next  choice  would  be  Raduchel 's  quantitative  measure. 
Unfortunately,  for  a  program  of  ACEIT' s  size,  it  would  be 
too  difficult  to  collect  the  data  needed  to  solve  his 
equation,  so  Simpson's  definition  of  user-friendliness,  such 
as  ease  of  use  and  documentation,  will  be  used. 
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III.  Methodology 


Introduction 

This  chapter  will  review  the  steps  taken  to  answer  the 
investigative  questions.  Each  section  of  the  chapter  is 
devoted  to  explaining  the  methodology  used  in  answering  a 
particular  investigative  question. 

How  User-Friendly  is  ACEIT? 

The  first  research  investigative  question  is: 

How  user-friendly  is  ACEIT?  Does  it  require  extensive 

training? 

The  information  provided  in  Chapter  I  and  Chapter  II 
will  be  used  as  the  background  against  which  to  evaluate 
ACEIT' s  user-friendliness.  This  will  include  evaluating 
both  the  internal  and  external  documentation  of  the  program. 
Additionally,  other  users  will  be  consulted  for  their 
opinions.  As  a  further  test  of  user-friendliness  and  the 
requirement  for  extensive  training,  this  researcher  will  not 
attend  the  ESD  training  course.  Therefore,  this  researcher 
will  receive  only  the  limited  training  offered  by  the  ACEIT 
tutorial,  the  User's  Guide,  and  discussions  with  other 
users.  A  one  day  trip  to  ESD/ACC  will  include  some  time  for 
training  if  needed. 

Part  of  the  answer  to  this  question  will  also  be 
provided  by  the  AFIT  Graduate  Cost  Analysis  students  who 
will  be  briefly  introduced  to  the  system  through  a  lesson 
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plan  developed  in  support  of  question  #5.  If  they  can 
"quickly  and  effectively"  be  introduced  to  ACEIT,  that  will 
support  the  concept  that  the  system  is  user-friendly  and 
does  not  require  extensive  training. 

Is  Cost  Estimating  Expertise  Required? 

The  second  research  investigative  question  is: 

Does  proper  use  of  ACEIT  require  cost  estimating 
expertise  or  does  it  prompt  even  a  novice  user  to 
generate  accurate  estimates? 

The  answer  to  this  question  was  partially  answered  by 
Randall  Bradley  in  his  evaluation  of  the  ACE  module  and 
recreation  of  an  historical  estimate  (4).  However,  he  used 
data  available  off-line  which  was  simply  input  and 
calculated  by  ACEIT.  In  other  words,  he  did  verify  that  the 
ACEIT  calculaLun  routines  work  properly,  but  he  did  not 
verify  that  a  novice  would  be  correctly  prompted  by  the 
system  to  make  effective  choices  using  the  entire  integrated 
system  (the  databases,  the  CERs,  and  STATPAK) .  Therefore, 
it  is  hoped  that  an  historical  ESD  estimate  can  be 
replicated  which  allows/requires  this  novice  estimator  to 
use  the  entire  system.  If  this  is  not  possible,  ESD  and 
AFIT  personnel  knowledgeable  about  ACEIT  will  be  interviewed 
as  to  their  opinions  on  this  question. 
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Is  STATPAK  Accurate  and  Reliable? 

The  third  research  investigative  question  is: 

Does  the  ACEIT  Statistics  Package  give  the  same 
estimates  and  statistics  as  the  Statistical  Analysis 
System  ( SAS ) ? 

This  question  will  be  answered  by  running  data  provided 
as  samples  in  the  ACEIT  User's  Guide  as  well  as  AFIT 
Graduate  Cost  Analysis  courses  through  both  STATPAK  and  SAS. 
Another  test  of  STATPAK 's  accuracy  will  be  to  run  the 
Longley  Data  Set  through  STATPAK  and  compare  the  results  to 
Longley's  answers.  Since  the  results  should  be  99.9%  the 
same,  any  differences  will  be  investigated  for  cause. 

As  an  expansion  of  this  question,  learning  curve  data 
will  also  be  run  through  Captain  Larry  D.  Hutchison's 
Microcomputer  Program  for  the  Solution  of  Learning  Curve 
Computations  (10) .  This  program,  written  as  a  thesis 
project  for  the  AFIT  Master  of  Science  in  Systems 
Management,  is  now  used  regularly  in  AFIT  courses. 

Answering  this  question  requires  a  working  knowledge  of 
STATPAK,  SAS,  and  the  microcomputer  learning  curve  program. 

Benefits  and  Problems  of  Air  Force  Implementation 

The  fourth  research  investigative  question  is: 

What  are  some  of  the  potential  benefits  and  problems  to 
be  avoided  in  successfully  implementing  ACEIT  in 
organizations  throughout  the  Air  Force? 

This  question  requires  a  working  knowledge  of  ACEIT  as 

well  as  some  exposure  to  problems  that  may  occur  in 

implementation.  This  researcher  will  base  an  answer  to  this 
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question  on  interviews  with  ESD  personnel  who  initiated 
ACEIT,  and  with  AFIT,  SSD,  and  BSD  individuals  who  have 
either  already  implemented  ACEIT  or  are  considering  it. 

Is  a  Quick  and  Effective  Introduction  Possible? 

The  last  research  investigative  question  is 

Can  ACEIT  be  quickly  and  effectively  introduced  to  the 
AFIT  Graduate  Cost  Analysis  students  who  are  unfamiliar 
with  the  system? 

This  question  will  be  addressed  by  developing  a  lesson 
plan  to  introduce  the  1989S  AFIT  Graduate  Cost  Analysis 
Class  to  ACEIT  as  part  of  their  Cost  Seminar.  Following 
development  of  the  lesson  plan,  it  will  be  taught  and 
feedback  sought  from  the  students.  The  objective  of  this 
introduction  is  not  to  make  ACEIT  experts  of  the  students 
but  rather  to  familiarize  them  with  the  system:  to  make  them 
aware  of  ACEIT  and  its  capabilities,  and  to  give  them  some 
limited  hands-on  experience.  The  goal  is  that  should  the 
students  encounter  ACEIT  at  their  ne-xt  assignment  (a  likely 
scenario)  they  will  be  familiar  with  it. 
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Introduction 


This  chapter  details  the  result  of  the  research 
regarding  each  investigative  question.  Each  question  is 
addressed  in  its  own  separate  section.  Conclusions  and 
recommendations  follow  in  Chapter  V. 

Question  1:  How  user-f riendlv  is  ACEIT  ... 

ACEIT  is  very  user-friendly  and  it  does  not  require 
extensive  training,  although  training  should  be  available  to 
those  who  desire  or  need  it. 

The  above  conclusion  is  based  not  just  on  the  personal 
experience  of  this  researcher,  but  also  on  the  opinions 
expressed  in  the  unpublished  reviews  of  ACEIT  and  on  how 
well  ACEIT  met  Simpson's  definition  of  user-friendliness  as 
outlined  in  Chapter  II. 

Personal  Experience. 

As  mentioned  earlier,  this  researcher  did  not  attend  a 
formal  ACEIT  training  course  but  instead  learned  the  system 
through  the  built-in  tutorial,  novice  learning  screens,  and 
exercises  provided  by  someone  who  had  attended  the  course. 
This  "training"  proved  sufficient  to  gain  a  working 
knowledge  of  the  system.  Although  a  brief  1-day  visit  with 
ESD  ACEIT  experts  definitely  refined  the  skills  needed  to 
fully  utilize  ACEIT,  a  working  knowledge  of  ACEIT  is 
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certainly  obtainable  from  the  system  itself.  An  extensive 
training  course  should  not  be  mandatory. 

One  can  learn  ACEIT  on  his  own  because  it  is  very  user- 
friendly.  It  would  be  nearly  impossible  to  obtain  a  working 
knowledge  of  a  system  as  involved  as  ACEIT  on  your  own  if  it 
were  not  user-friendly.  Throughout  the  process  of  building 
an  estimate,  ACEIT  will  prompt  the  user,  even  in  the  expert 
mode,  if  certain  elements  are  missing  or  incorrectly 
entered.  One  can  exit  the  system  at  any  time  or  go  to  a 
help  screen  for  further  guidance.  The  system  allows  a  user 
to  easily  correct  mistakes  and  proceed  on  with  the  estimate. 
All  in  all,  it  was  very  easy  to  use. 

Other  Opinions . 

Birkofer  (3)  and  Bradley  (4)  confirmed  this 
researcher’s  diagnosis  that  ACEIT  was  very  user-friendly. 
Birkofer  briefly  reviewed  each  module  of  ACEIT  and  concluded 
that  each  was  either  "easy"  or  "very  easy"  to  use  (3). 
Bradley  limited  his  review  to  the  ACE  module. 

Unfortunately,  he  had  some  difficulties  initially  using  the 
system  (4:23).  However,  these  difficulties  were  not  really 
with  ACEIT  but  instead  with  its  configuration  at  AFIT.  The 
difficulties  Bradley  experienced  in  accessing  the  system 
were  the  same  as  those  documented  and  experienced  by  the 
1988S  AFIT  Graduate  Cost  Analysis  Class.  These  problems, 
largely  due  to  the  use  of  a  Z-248  microcomputer  as  a 
terminal,  have  been  eliminated  by  the  addition  of  numerous 
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VT-100  terminals  (the  preferred  terminal  to  use  with  ACEIT) 
as  well  as  a  new  release  of  ACEIT  (and  the  User's  Guide) 
which  has  improved  documentation  for  using  the  system  with  a 
different  terminal  (such  as  a  Zenith  PC) .  Although  the 
system  could  always  be  used  with  different  terminals, 
different  keystrokes  were  required  and  the  initial 
documentation  was  hard  to  follow  if  a  VT-100  was  not  being 
used.  This  created  a  frustrating  environment  where  mistakes 
were  made,  not  because  they  were  actual  estimating  mistakes 
but  because  the  incorrect  key  was  being  used.  Even  with  the 
difficulties  Bradley  experienced,  he  concluded  that  ACEIT 
"on  the  whole,  is  fairly  easy  to  use"  (4:25). 

One  other  comment  Bradley  made  regarding  the  system 
should  also  be  addressed  here.  In  his  recommendations, 
Bradley  stated  that  ACEIT  was  "suited  mainly  for  one  person 
accomplishing  an  entire  systems  estimate.  Although  it  could 
be  possible  to  incorporate  this  model  in  a  team  estimating 
environment"  (4:25).  In  contrast  to  Bradley's  conclusion 
that  ACEIT  is  mainly  suited  for  one  estimator,  is  evidence 
that  ACEIT  works  very  nicely  with  team  estimates.  According 
to  Ms  Margaret  Weech,  ESD/ACC  ACEIT  Program  Manager,  ESD  has 
recently  completely  an  Independent  Cost  Estimate  (ICE)  on 
the  Cheyenne  Mountain  Upgrade  using  ACEIT  (23).  This  large 
estimate  was  definitely  a  team  estimate.  Although  the 
current  version  of  ACEIT  does  not  permit  merging  ACE  files 
(as  a  future  version  will),  ESD  gave  the  estimators  access 
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to  an  account  holding  only  the  Cheyenne  estimate.  This 
allowed  estimators  to  work  on  the  estimate  within  that 
account  or  copy  it  into  their  own  account  and  then  manually 
input  final  calculations  into  the  main  account.  The  ability 
to  support  team,  as  well  as  individual,  estimates  is  more 
evidence  of  ACEIT's  user-friendliness. 

The  1989  AFIT  Graduate  Cost  Analysis  Class  was  able  to 
experience  ACEIT  from  the  beginning  without  the  frustrating 
terminal  difficulties  experienced  by  the  previous  class  and 
Bradley.  This  class  was  an  interesting  test  group  because 
they  came  from  varied  cost  and  computer  backgrounds.  Some 
individuals  were  extremely  computer  literate,  others  had 
almost  no  computer  experience.  The  same  was  true  for  their 
cost  estimating  experience.  Some  individuals  entire  cost 
estimating  background  was  limited  to  their  recent  academic 
courses,  others  were  considered  experienced  estimators 
before  attending  the  AFIT  program. 

This  difference  in  background  as  well  as  the  briefness 
of  the  ACEIT  introduction  was  definitely  a  test  of  ACEIT's 
user-friendliness.  Evidence  of  ACEIT's  user-friendliness  is 
that  with  very  little  instruction  the  students  were  able  to 
complete  an  entire  full  scale  engineering  development  cost 
estimate.  However,  computer  problems  during  the  initial 
hands-on  lesson,  long  delays  caused  by  a  heavy  load  on  the 
time-shared  computer  (not  necessarily  the  ACEIT  class),  and 
the  depth  of  the  estimating  exercise  made  the  time  available 
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really  inadequate.  Based  on  the  survey  of  the  class 
(Appendix  E)  several  students  requested  the  introduction  be 
lengthened.  Others  requested  a  review  of  the  novice  mode 
before  doing  the  estimate  (although  this  was  available  to 
them  to  review  on  their  own) .  The  students  did  not  have  a 
chance  to  experiment  with  the  system  in  class  before 
attempting  the  estimate.  Even  with  these  drawbacks,  the 
overall  consensus  was  that  ACEIT  was  somewhat  user-friendly. 
Additionally,  several  students  commented  they  felt  the 
system  was  very  user-friendly  and  easier  to  use  than  they 
had  originally  thought  it  would  be. 

According  to  Simpson  . .  . 

ACEIT  definitely  meets  Simpson's  requirements  for  being 
user-friendly.  As  already  detailed  above,  ACEIT  is  "easy  to 
use,  tolerant  of  operator  errors,  easy  to  learn,  and 
acknowledges  that  human  beings  are  imperfect  creatures" 
(21:2)  . 

The  exceptionally  clear  and  easy  to  understand  ACEIT 
internal  and  external  documentation  also  meets  Simpson's 
requirements.  The  current  release  (version  2.5)  and  current 
User's  Guide  are  excellent  examples  of  what  Simpson 
considers  user-friendly.  The  system  is  designed  to  grow 
with  the  user  by  providing  novice  and  expert  modes.  The 
internal  documentation  prompts  respond  according  to  the  mode 
being  used.  The  User's  Guide  is  designed  with  several 
sections  for  the  novice  user.  The  ACEIT  Introduction,  the 
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ACE  Tutorial,  and  STATPAK  appendix  are  specifically  geared 
toward  the  novice  user  and  probably  would  not  be  referenced 
by  an  expert  user  at  all.  However,  the  guide  is  still 
valuable  for  those  with  a  lot  of  estimating  and/or  ACEIT 
experience . 

In  summary,  ACEIT  appears  to  have  been  designed  from 
the  very  beginning  to  be  user-friendly  by  having  various 
modes  with  appropriate  documentation  provided  for  users  of 
different  levels.  An  intermediate  mode  is  planned  but  not 
currently  available.  Additionally,  based  on  discussions 
with  ESD  ACEIT  personnel,  Tecolote  Research,  Inc  has  been 
very  responsive  to  making  requested  (and  funded)  user- 
friendly  improvements  to  the  system  (5). 

Question  2  -  Is  Cost  Estimating  Expertise  Required  . . . 

Some  cost  estimating  experience  is  necessary  to  be  able 
to  effectively  use  ACEIT,  understand  the  User's  Guide,  and 
correctly  select  the  appropriate  models  or  cost  factors  for 
each  WBS  item.  However,  it  is  this  researcher's  opinion 
that  extensive  expertise  is  not  required  because  the  system 
does  an  excellent  job  of  prompting  a  novice  user. 

An  initial  goal  in  attempting  to  answer  this  question 
was  for  this  novice  estimator  to  replicate  an  existing  cost 
estimate  from  scratch  using  the  ACEIT  system.  However, 
discussions  with  ESD  Cost  personnel  quickly  revealed  that 
this  was  not  possible  (5).  Time  constraints,  a  lack  of 
knowledge  of  ESD  programs,  a  lack  of  knowledge  of  ESD 
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engineering  personnel,  physical  distance  from  ESD,  and 

changes  in  data  over  time  all  contributed  to  making  this 

initial  goal  unattainable  (5) . 

However,  while  this  researcher  was  unable  to  replicate 

an  estimate  from  scratch,  as  mentioned  earlier  Randall 

Bradley  was  able  to  accomplish  a  replication  of  ASD  LIDAR  by 

using  available  LIDAR  cost  data.  This  was  different  from 

replicating  the  estimate  from  scratch.  Bradley  did  not 

search  ACEIT's  databases  for  analogous  programs  or  use 

ACEIT's  models.  He  simply  input  appropriate  costs  indicated 

from  LIDAR  data  or  off-line  runs,  or  throughput  equations 

such  as  ASD/RW  Factors  14:117).  The  end  result  was  that  he 

validated  the  computational  capabilities  of  the  ACE  module 

of  ACEIT.  When  the  same  numbers  were  input  into  ACEIT  the 

same  estimate  was  generated.  Referring  to  the  estimate  in 

base  year  dollars  Bradley  stated 

The  cost  estimate  produced  by  ACE  matched  (WBS  element 
by  WBS  element)  the  estimate  I  used  as  input.  It 
should  be  noted  that  this  was  an  expected  outcome  since 
all  of  my  inputs  were  either  direct  throughputs  or 
equations  set  up  to  produce  specific  answers.  (4:16) 

An  interesting  result  of  Bradley's  estimate  was  that  he 

validated  the  inflation  calculations  by  identifying  an  error 

made  in  the  original  LIDAR  then  year  dollar  estimate. 

The  Then  Year  estimate  which  ACE  calculated  did  not 
match  the  ASD/AE  LIDAR  estimate  that  I  had  used  as 
input.  I  traced  the  steps  used  in  computing  the  ASD/AE 
estimate  and  concluded  that  his  methodology  was 
incorrect.  After  contacting  the  LIDAR  estimator,  he 
confirmed  that  his  methodology  was  in  error  and  that 
ACE  computed  the  Then  Year  estimate  correctly.  (4:19) 
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So  it  can  be  concluded  that  given  all  the  appropriate 

methodology  a  novice  estimator  can  certainly  use  ACEIT  to 

create  an  effective  estimate.  However,  the  question  remains 

of  whether  a  novice  estimator  can  use  ACEIT  to  create  an 

estimate  from  scratch  (where  the  estimator  would  have  to 

determine  the  appropriate  methodology  to  use) . 

ESD  Cost  personnel  indicated  the  system  was  not 

intended  to  be  used  by  individuals  with  cost  expertise, 

but  instead  that  some  expertise  was  required  (5,  22).  The 

ACEIT  User's  Guide  does  not  specifically  mention  cost 

expertise  but  does  comment 

Though  tailored  specifically  to  the  ESD/ACC 
requirements,  it  is  intended  that  this  software  be  used 
by  trained  Program  Office  personnel  to  develop  and 
maintain  Program  Office  cost  estimates  in  a  manner 
totally  consistent  with  accepted  cost  analysis 
standards,  procedures,  and  formats.  (13: Sec  2,1) 

The  guide  does  not  define  what  exactly  is  meant  by  "trained" 

personnel  (whether  they  are  trained  to  operate  ACEIT  or 

trained  in  the  area  of  cost  estimation) .  AFIT  faculty 

indicated  that  ACEIT  could  be  dangerous  if  used  by  totally 

inexperienced  personnel  and  that  some  cost  estimating 

expertise  was  required  (15)  .  Captain  Chris  Voss  (SSD)  even 

commented  that  more  then  "some"  cost  estimating  expertise 

was  required  (22) . 

The  reason  some  cost  expertise  is  required  can  be 
traced  back  to  the  fact  that  ACEIT  is  not  an  expert  system. 
As  mentioned  in  Chapter  I,  ACEIT  will  not  tell  a  user  what 
methodology  to  use.  It  may  suggest  alternatives,  but  the 
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user  must  determine  which  methodology  is  appropriate.  ACEIT 
does  not  prevent  an  inexperienced  estimator  from  making  the 
mistakes  noted  in  Kazanowski’s  article  described  in  Chapter 
II  (12).  An  estimator  can  still  miss,  misuse,  or  ignore 
pertinent  criteria.  Additionally,  the  system  will  not 
prevent  an  estimator  from  extrapolating  beyond  the  relevant 
range.  The  system  does  provide  comments  as  to  what  the 
relevant  range  is  but  the  estimator  can  choose  to  ignore 
those  comments. 

ACEIT  will  provide  support  to  an  estimator.  The  CERs 
linked  to  specific  WBS  items,  the  definitions  provided  and 
the  checks  within  ACEIT  which  identify  missing  variables  all 
are  geared  to  guide  an  analyst.  However,  the  information 
provided  within  ACEIT  is  not  complete,  it  is  more  like  an 
executive  summary.  Before  including  a  model  or  CER  in  an 
estimate,  the  estimator  should  first  check  the  source 
documentation  to  ensure  the  model  or  CER  is  appropriate. 
Estimating  with  or  without  ACEIT,  the  same  rule  applies:  an 
estimator  must  ensure  his  methodology  fits  the  situation. 

For  a  novice  estimator,  this  can  be  confusing.  ACEIT' s 
sometimes  brief  documentation  may  not  be  sufficient  to  alert 
an  inexperienced  estimator  to  a  potential  problem.  For 
instance,  ACEIT  includes  two  software  cost  estimating 
models:  COCOMO  and  WICOMO.  If  the  same  data  is  input  into 
both,  different  answers  will  result.  Neither  the  ACEIT 
User's  Guide  nor  the  internal  documentation  make  clear  that 
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the  Tecolote  COCOMO  program  is  an  uncalibrated  version  of 
the  intermediate  COCOMO  equations  and  the  Wang  WICOMO 
program  is  an  uncalibrated  version  of  the  detailed  COCOMO 
equations.  This  novice  estimator  originally  assumed  the  two 
programs  would  provide  the  same  estimates  as  REVIC  (another 
COCOMO  program  -  a  calibrated  version  of  the  intermediate 
version)  taught  at  AFIT.  It  was  only  after  discussions  with 
Capt  Joe  Dean  of  ESD/ACC  that  the  differences  between  the 
programs  were  clarified  (7).  This  type  of  mistake  could 
easily  be  made  by  a  novice  estimator  given  the  current 
internal  and  external  documentation  in  ACEIT.  However,  if  a 
novice  estimator  takes  it  upon  himself  to  investigate 
everything  he  uses  and  not  wait  for  the  system  to  alert  him, 
ACEIT  can  be  a  great  guide.  A  novice  must  always  remember 
that  ACEIT  will  not  provide  all  the  answers. 

Therefore  in  conclusion,  ACEIT  is  designed  to  prompt 
novice  estimators  to  help  them  make  accurate  estimates. 
However,  some  cost  estimating  expertise  is  required  since 
ACEIT  is  not  an  expert  system.  It  will  aid  an  estimator, 
but  the  estimator  still  must  make  the  critical  choice  of 
using  the  most  appropriate  estimating  methodology. 

Question  3  -  Is  STATPAK  Accurate  and  Reliable? 

The  ACEIT  statistical  analysis  module,  STATPAK,  appears 
to  be  just  as  accurate  and  reliable  as  SAS  or  the  Hutchison 
Learning  Curve  program.  Additionally,  although  not  as 
powerful  as  SAS,  STATPAK  is  definitely  easier  to  use  and 
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appears  to  have  all  the  essential  computational 
capabilities . 

Four  different  sets  of  data  were  used  to  compare  the 
statistical  analysis  capabilities  of  STATPAK  and  SAS .  The 
first  set  was  the  practice  data  used  as  an  example  in  the 
STATPAK  section  of  the  ACEIT  User's  Guide  (Appendix  G) .  The 
second  set  contained  actual  helicopter  engine  performance 
data  but  the  engine  cost  data  was  modified  to  maintain 
proprietary  data  rights  (Appendix  G) .  The  third  set 
contained  actual  rocket  engine  performance  data  but  again 
the  cost  data  was  modified  to  maintain  proprietary  data 
rights  (Appendix  G) .  Both  the  second  and  third  data  sets 
were  provided  by  AFIT  faculty  and  are  used  in  AFIT  cost 
classes.  The  last  data  set  run  was  the  Longley  data  set. 
This  data  was  run  on  both  STATPAK  and  SAS  and  the  answers 
were  compared  to  Longley 's  results. 

The  same  linear  and  log  linear  regression  equations  and 
statistics  were  calculated  by  both  SAS  and  STATPAK  for  the 
first  three  data  sets;  nonlinear  regressions  were  not 
investigated.  Sample  outputs  are  provided  at  Appendices  A 
and  B.  However,  STATPAK  rounds  all  figures  to  four  decimal 
places  while  SAS  carries  the  equations  to  eight  decimal 
places.  In  some  cases  the  STATPAK  figures  differed  in  the 
fourth  decimal  place,  or  for  very  large  (eight  digit) 
numbers  in  the  units  column.  These  differences  are 
negligible.  Since  the  figures  were  99.9%  the  same  (except 


40 


as  noted  below) ,  the  programs  produce  essentially  identical 
answers . 

STD  RES  on  the  STATPAK  output  appears  to  be  the  only 
confusing  statistic.  According  to  the  User's  Guide,  this  is 
the  standardized  residual  and  it  is  described  as  the 
residual  divided  by  its  standard  deviation  (13:App  A, 17), 
Neter,  Wasserman,  and  Kutner,  in  Applied  Linear  Regression 
Models ,  define  a  standardized  residual  as  the  residual 
divided  by  the  square  root  of  the  mean  square  error;  while 
they  define  a  studentized  residual  as  the  residual  divided 
by  its  standard  deviation  ( 18 ; 404-405 ) ,  SAS  produces 
figures  for  its  studentized  residual  which  are  at  least 
97.4%  the  same  as  STATPAK  produces  for  its  standardized 
residual.  It  appears  that  the  figures  aren't  99.9%  the  same 
due  to  the  fact  that  the  SAS  figures  are  carried  to  four 
decimal  places  versus  only  two  for  STATPAK.  If  the  SAS 
figures  were  rounded  to  two  decimal  points,  the  figures 
would  be  100%  the  same.  Since  the  abbreviation  STD  could  be 
for  standardized  or  studentized,  the  STATPAK  documentation 
and  printouts  should  clarify  what  STD  RES  stands  for  and  use 
a  conventional  name  for  the  residual. 

Another  confusing  characteristic  of  STD  RES  is  how  it 
handles  an  undefined  value.  Observation  6  on  the  sample  SAS 
and  STATPAK  outputs  has  a  residual  value  of  zero  as  well  as 
a  standard  error  of  the  residual  value  of  zero.  Therefore, 
the  STD  RES  calculation  (dividing  by  zero)  is  undefined. 
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SAS  simply  prints  a  dot  out,  but  STATPAK  prints  out  10,000. 
This  is  not  explained  in  the  ACEIT  User's  Guide,  but  10,000 
appears  to  be  a  default  value.  A  more  conventional  notation 
such  as  a  dot  or  an  asterisk  should  probably  be  used. 

An  interesting  result  developed  when  the  Longley  data 
was  run:  neither  SAS  nor  STATPAK  gave  the  identical  results 
that  Longley  calculated.  The  comparison  of  results  is 
presented  in  Table  1.  In  contrast  to  the  parameter 
estimates  Longley  developed  when  testing  the  data  on  various 
proarams,  at  least  both  SAS  and  STATPAK  always  had  the 
correct  sign  and  at  least  were  the  right  magnitude.  In 
other  words  they  were  both  in  the  ball  park.  Since  an  in- 
depth  analysis  of  the  calculations  used  by  SAS  and  STATPAK 
in  computing  the  least  squares  regression  equations  is 
beyond  the  scope  of  this  research,  the  only  conclusion  that 
can  be  made  is  that  SAS  and  STATPAK  produce  comparable 
results . 
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Table  1 


Regression  Para.meters  Estimated  Usino  Lonaley  Data 


LONGLEY 

(14:332) 


SAS 


3TATPAK 


II'TEP.CEPT  -3  4' 
XI 


X3 

X4 

X5 

X6 


2253 .5330 
15.0513722'7 
-0.03581917 
-2.02022980 
-1,03322636 
-0.05110410 
1329.15146461 


-3493333. 

25 . 3750'7003 
-0.0389083 
-2.0561-7 
-1.04133 
-0.0269374 
1333.45774 


-3423915.0000 

14.4951 

-0.0343 

-1.9971 

-1.0264 

-0.055"' 

1301.3125 
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STATPAK  was  much  easier  to  use  than  SAS.  The  user  does 


not  have  to  remember  specific  commands  and  their  peculiar 
formats  or  create  a  program  to  produce  an  output.  STATPAK 
asks  the  user  in  english  what  output  he  desires  and  it  is 
simply  produced. 

The  only  capability  STATPAK  lacked  which  might  be 
desirable  is  that  of  running  correlation  matrices  on  a  set 
of  variables.  STATPAK  will  however,  provide  scatterplots  of 
the  variables  specified. 

STATPAK  also  produced  learning  curve  results  comparable 
to  those  produced  by  the  Hutchison  program.  Two  different 
data  sets  provided  by  AFIT  faculty  were  used  to  compare  the 
two  programs  (Appendix  G) .  Both  weighted  and  unweighted 
unit  curve  calculations  were  made.  Sample  outputs  are 
provided  at  Appendices  C  and  D.  Again,  the  last  digit  was 
different  in  some  cases  but  this  could  be  due  to  one  program 
running  on  a  microcomputer  and  one  running  on  a  mainframe 
computer  which  may  result  in  different  rounding  of  the 
calculations.  An  example  is  Data  Set  2:  1)  STATPAK 
generated  a  learning  curve  with  Unit  1  cost  of  1783.923  and 
slope  of  82.73%,  while  2)  Hutchison  generated  a  learning 
curve  with  Unit  1  cost  of  1784.13  and  slope  of  82.72%. 

These  figures  are  99.9%  the  same.  Even  with  the  slight 
difference  in  unit  one  cost  and  slope  the  two  programs 
resulted  in  comparable  predicted  costs  for  a  future  lot. 
Using  their  respective  parameters  to  predict  a  new  lot  for 
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units  252-311:  1)  ACE  using  STATPAK  parameters  predicted  a 
lot  cost  of  22900.2,  2)  Hutchison  predicted  a  lot  cost  of 
22889.54,  with  95%  confidence  interval  limits  of  21590.00  - 
24267.30  (aaain  the  figures  are  99.9%  the  same). 

It  should  be  noted  that  for  doing  a  lot  of  learning 
curve  analysis  (worlcing  with  deletions,  additions,  or  breaks 
in  production)  the  Hutchison  program  is  more  flexible  than 
STATPAK.  The  Hutchison  program  contains  a  Calculator  option 
which  allows  the  user  to  easily  vary  first  unit  value, 
slope,  given  unit/lot,  or  to  change  from  unit  to  cumulative 
average  formulation  (or  vice-versa)  (10:61). 

Question  4  -  Benefits/Pitfalls  of  Air  Force  Implementation 

This  researcher  believes  that  overall  the  Air  Force¬ 
wide  implementation  of  ACEIT  will  be  good.  It  should  be 
made  clear  that  at  this  point  in  time  the  term  "Air  Force¬ 
wide  implementation"  refers  only  to  research  and  development 
organizations.  It  is  not  this  researcher's  intent  to  infer 
ACEIT  should  be  implemented  at  every  base  in  the  Air  Force. 
However,  even  though  there  are  specific  benefits  to  be 
gained  from  this  implementation  there  are  also  pitfalls  to 
be  avoided  or  the  implementation  will  be  a  failure. 

Benef  it  s_^ 

Air  Force-wide  implementation  of  ACEIT  has  numerous 
benefits  which  should  make  it  very  helpful  to  cost 


estimators . 


As  discussed  in  Chapter  I,  ACEIT  is  essentially  an 
architecture  and  as  such  can  be  tailored  to  fit  specific 
organizational  requirements.  This  is  the  beauty  of  the 
ACEIT  system.  Even  though  the  system  was  originally 
designed  for  ESD.  It's  databases,  WBSs,  CERs ,  and  models 
can  be  exchanged  for  those  of  interest  at  any  organization. 
For  instance,  Space  Systems  Division  (SSD)  has  already 
included  its  own  WBS  with  links  to  CERs  "blessed"  by  SSD/ACC 
for  use  in  estimating  (22).  In  addition,  a  Space  Systems 
Division  specific  model  has  been  added  with  plans  to  add 
additional  models  in  the  future  (22). 

As  far  as  databases  are  concerned,  efforts  are  underway 
to  standardize  the  format  of  data  which  may  be  of  use  to 
several  organizations  (22) .  This  would  mean  that  common 
data  could  be  collected  in  one  location  and  yet  be  usable  by 
numerous  organizations. 

With  many  organizations  using  the  same  ACEIT  program, 
organizations  could  also  share  data.  Since  the  data  formats 
would  be  the  same,  data  collected  and  normally  used  at  one 
location,  could  easily  be  transferred  to  another 
organization  and  used  on  their  ACEIT  system. 

Another  benefit  of  organizations  using  a  common  program 
is  that  personnel  transferring  between  different 
organizations  will  not  have  to  learn  a  new  system. 

Personnel  will  already  be  familiar  with  the  system.  The 
ACEIT  output  may  also  be  a  benefit.  With  the  common  ACEIT 
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output,  estimators  should  have  an  easier  time  understanding 
each  other's  work.  Additionally,  the  ACEIT  output  format 
helps  to  provide  a  clear  history  of  the  estimate  with  ample 
room  for  references  and  notepad  comments. 

Pitfalls . 

Unfortunately  the  beauty  of  the  ACEIT  system  (being 
able  to  fill  the  system  with  organization  specific  data)  may 
also  be  its  major  potential  pitfall.  The  data  input  at  each 
location  must  be  carefully  screened  in  terms  of  its 
appropriateness  and  in  terms  of  measurement  and 
typographical  errors.  In  this  case,  "data"  includes  data, 
CERs ,  and  models.  For  instance,  the  unique  WBS  and 
corresponding  CERs  linked  to  those  WBS  items  must  be  very 
carefully  examined  (5,2).  At  ESD,  the  Technical  Director  of 
Cost  Analysis  personally  reviewed  each  CER  linked  to  a  WBS 
item  as  well  as  the  text  describing  each  CER  (5).  Only  CERs 
or  factors  agreed  upon  by  the  senior  analysts  in  ACC  were 
included  as  WBS  linked  CERs  (5).  BSD  is  also  carefully 
screening  their  CERs  (9) .  Although  ESD  plans  to  maintain 
configuration  control  of  the  ACEIT  program  itself,  they 
cannot  possibly  maintain  configuration  control  of  the  unique 
data  input  into  the  system  at  each  location  (5).  This  will 
be  the  responsibility  (most  likely)  of  ACC  at  each  location. 
To  maintain  high  quality  estimates  with  ACEIT,  requires  that 
the  data  initially  input  into  ACEIT  be  high  quality  and  that 
the  data  be  continuously  updated  to  avoid  obsolescence. 
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Training  is  another  potential  pitfall  for  ACEIT. 
Although  the  system  is  user-friendly  and  it  is  not  difficult 
to  gain  a  working  knowledge  of  ACEIT,  some  individuals  may 
desire  or  reguire  additional  training.  ESD  currently  hosts 
5-day  training  workshops  taught  by  ESD  personnel  (5).  SSD 
on  the  other  hand  has  contracted  out  to  have  Tecolote 
Research  Inc  provide  a  training  workshop  (22).  Whatever  the 
situation,  some  training  should  be  available  to  those  who 
need  it  or  the  ACEIT  system  will  not  be  used  effectively. 

An  area  which  may  also  be  a  pitfall  if  not  carefully 
controlled  is  that  of  proprietary  data.  User  access  to  the 
various  data  sets  or  models  which  ACEIT  includes  must  be 
carefully  controlled  so  that  only  authorized  personnel  have 
access  to  proprietary  data.  The  proprietary  data  issue  must 
be  considered  if  contractors  are  to  have  access  to  the  ACEIT 
system.  For  instance,  TRW  works  closely  with  BSD  personnel. 
According  to  Mr  David  Hanson,  Chief  of  Cost,  BSD/ACC,  TRW 
has  a  nondisclosure  clause  in  their  contract  (9). 
Additionally,  BSD  is  only  including  data  in  their  database 
which  TRW  may  access  (9) .  At  ESD,  task  order  contractors 
may  not  access  other  contractor’s  proprietary  CERs  (23). 

Question  5  -  Is  a  Quick  and  Effective  Introduction  Possible? 

It  is  possible  to  guickly  and  effectively  introduce 
ACEIT  to  AFIT  Graduate  Cost  Analysis  students. 

In  contrast  to  the  five-day  course  being  offered  at  ESD 
and  the  two  and  a  half  day  course  being  offered  at  SSD,  this 
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brief  introduction  only  included  four  one-and-a-quarter-hour 
periods.  The  introduction  was  not  intended  to  accomplish 
the  same  level  of  detail  presented  in  longer  courses,  but  it 
was  still  a  challenge  to  introduce  the  subject  in  the 
significantly  reduced  time-frame.  The  goal  of  the 
introduction  was  to  familiarize  the  students  with  ACEIT. 
Although  a  simple  briefing  could  have  informed  them  of 
ACEIT 's  capabilities,  this  researcher  considered  some  hands- 
on  experience  mandatory.  Therefore,  only  one  out  of  four 
periods  were  in  the  lecture  format.  The  first  period  was 
used  to  brief  the  students  on  ACEIT,  its  capabilities,  and 
what  to  expect  in  the  next  three  periods.  The  second, 
third,  and  fourth  periods  provided  hands-on  training  and 
guidance.  Each  student  was  to  follow  through  the  well 
documented  Tutorial  in  the  ACEIT  User's  Guide. 

Unfortunately,  computer  problems  during  the  first  period 
resulted  in  only  two  students  being  able  to  use  their 
terminal  while  the  other  seven  students  watched.  This 
resulted  in  very  little  learning.  However,  during  the  next 
two  class  periods,  students  were  able  to  work  through  an 
estimate  based  on  an  ESD  training  exercise  {Appendix  F) . 

Several  lessons  were  learned  from  this  experience. 
First,  the  ACEIT  Tutorial  is  an  excellent  learning  tool. 
However,  the  size  and  diversity  of  the  tools  available  with 
ACEIT  make  it  very  difficult  to  cover  the  Tutorial  in  one 
classroom  period.  Additionally,  as  evident  from  the 
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students'  answers  to  the  questionnaire  (Appendix  E) , 
computer  background  and  cost  estimating  experience  play  a 
part  in  how  quickly  the  ACEIT  system  is  grasped.  Although 
the  hands-on  experience  reinforces  what  ACEIT  does,  it  also 
requires  the  students  to  become  adept  at  moving  around 
ACEIT.  Just  learning  to  use  the  ACEIT  commands  takes  some 
time  and  is  more  difficult  for  students  with  little  computer 
expertise.  Before  covering  the  Tutorial,  some  time  should 
be  devoted  to  just  explaining  some  of  the  basic  ACEIT 
commands.  This  would  also  help  students  learn  how  to 
recover  if  they  hit  the  wrong  command.  Also  a  reference 
card  with  basic  commands  for  moving  around  ACEIT  should 
probably  be  distributed.  The  cost  estimating  experience 
helped  students  understand  the  flow  of  steps  needed  to 
complete  the  estimate  with  ACEIT.  Those  with  less  cost 
estimating  experience  needed  more  time  and  more  direction  to 
accomplish  the  estimating  exercise.  These  students  probably 
could  have  worked  through  the  estimate  themselves  using  the 
novice  mode,  but  the  pace  of  the  brief  introduction  did  not 
allow  enough  time  to  work  through  the  novice  mode  in  class. 

Although  the  students  were  introduced  to  ACEIT,  if  an 
estimate  is  going  to  be  accomplished,  more  time  must  be 
allotted  to  ensure  adequate  understanding  of  the  many  facets 
of  this  complex  system.  Additionally,  even  with  an  expanded 
schedule  it  is  recommended  that  the  exercise  be  shortened 
somewhat.  The  exercise  used  and  a  recommended  shortened 
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version  are  included  in  Appendix  F.  A  suggested  expanded 
lesson  schedule  is: 

Lesson  1  -  Briefing  with  computer  demonstration. 

Lesson  2  -  Hands-on.  Basic  ACEIT  commands  and 

Tutorial  through  the  WBS. 

Lesson  3  -  Hands-on,  Finish  Tutorial. 

Lesson  4  -  Hands-on.  Start  exercise. 

Lesson  5  -  Hands-on.  Finish  exercise. 

Another  lesson  learned,  is  to  have  alternate  plans  if 
the  computer  equipment  fails  to  work  properly.  The  hands-on 
experience  is  essential,  but  since  classes  cannot  just  be 
canceled  due  to  computer  malfunctions,  backup  plans  should 
be  available. 

A  last  lesson  learned  would  be  to  introduce  the  ACEIT 
system  earlier  in  the  graduate  program.  The  timing  of  this 
particular  introduction  unfortunately  coincided  with  thesis 
preparation  (which  definitely  took  priority  in  the  students' 
minds) .  With  a  heavy  thesis  load,  students  are  unlikely  to 
spend  significant  time  on  ACEIT  outside  of  class.  To  be 
really  effective  during  the  class,  the  students  should  at 
least  read  through  the  Tutorial  and  exercise  ahead  of  time. 
Another  benefit  of  introducing  the  system  earlier  in  the 
program  is  that  students  could  then  use  it  in  support  of 
some  of  their  other  classes  and  become  even  more  familiar 
with  it  as  they  did. 
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V.  Conclusions  and  Recommendations 


Introduction 

This  chapter  summarizes  the  conclusions  identified  in 
Chapter  IV  and  makes  some  recommendations  regarding  further 
research  and  ACEIT  documentation. 

Conclusions 

The  ACEIT  system  is  very  user-friendly.  Both  the 
internal  and  external  documentation  are  easy  to  follow.  The 
availability  of  both  an  expert  and  a  novice  mode  tailors  the 
system's  prompts  to  the  user's  ability.  The  documentation 
is  so  good  that  it  enables  a  user  to  obtain  a  basic  working 
knowledge  of  the  system  without  extensive  training. 

However,  to  fully  utilize  all  of  ACEIT's  capabilities 
quickly,  training  should  be  available  for  those  that  desire 
or  need  it. 

Although  the  system  is  user-friendly,  it  still  requires 
some  cost  estimating  expertise.  ACEIT  is  not  an  expert 
system.  It  does  not  make  decisions  for  the  cost  estimator; 
it  only  provides  support.  A  user  must  have  some  background 
in  cost  estimating  to  fully  utilize  ACEIT's  capabilities  and 
to  be  able  to  wisely  select  from  any  options  ACEIT  may 
provide.  Even  the  novice  screens  will  only  support  an 
estimator  who  already  has  some  experience. 

The  STATPAK  module  appears  to  be  at  least  as  accurate 
as  SAS  and  the  Hutchison  Learning  Curve  Program. 
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Essentially  the  same  estimates  and  statistics  were  produced 
by  STATPAK  and  the  other  programs.  When  the  Longley  data 
set  was  used  STATPAK  and  SAS  produced  essentially  the  same 
parameter  estimates  although  neither  matched  the  results 
identified  by  Longley. 

The  major  benefit  of  ACEIT  may  also  be  it's  major 
potential  pitfall  in  implementing  the  system  throughout  the 
Air  Force  research  and  development  community.  The  fact  that 
ACEIT  is  a  framework  which  can  be  filled  with  organization 
unique  data  puts  a  great  responsibility  on  organizations  to 
quality  control  that  data.  ESD  will  maintain  quality 
control  of  the  ACEIT  program  itself.  However,  each 
organization  must  ensure  the  work  breakdown  structures, 
data,  and  cost  estimating  relationships  put  into  ACEIT' s 
databank  are  initially  accurate  and  accurately  documented, 
as  well  as  being  accurately  updated  on  a  continual  basis. 

Results  of  the  brief  introduction  of  AFIT  Graduate  Cost 
Analysis  students  to  ACEIT  indicate  that  ACEIT  can  be 
quickly  and  effectively  introduced.  This  does  not  mean  that 
the  AFIT  students  are  now  ACEIT  experts,  they  have  only  been 
introduced  to  the  system.  The  introduction  was  a 
combination  of  a  briefing  and  hands-on  experience  using 
ACEIT  to  complete  the  Tutorial  and  an  estimating  exercise. 
However,  the  short  four-period  introduction  was  too  fast 
paced  for  those  with  little  computer  experience  or  little 
cost  estimating  experience.  An  expanded  introduction  would 
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provide  more  time  to  explore  ACEIT's  capabilities.  However, 
even  an  expansion  to  five  periods  could  not  provide  the 
level  of  detail  currently  offered  in  the  ESD  five-day 
course.  The  five-period  introduction  would  still  be  an 
introduction . 

Recommendations 

Following  from  the  discussion  of  the  ACEIT  introduction 
to  the  AFIT  students,  the  first  recommendation  is  to  expand 
the  number  of  lessons  from  four  to  five,  and  reduce  the 
complexity  of  the  estimating  exercise.  This  will  give  the 
students  more  time  to  work  with  ACEIT.  It  is  not 
recommended  that  the  introduction  go  to  a  briefing  format 
only.  The  hands-on  experience  is  essential  if  the  students 
are  to  fully  grasp  the  potential  of  the  ACEIT  system. 

Throughout  this  research  a  few  recommendations  surfaced 
for  improvements  to  ACEIT  and  to  the  User’s  Guide.  One 
additional  computational  capability  desired  is  to  be  able  to 
input  the  inflation  indices  used  in  calculating  the  then 
year  estimates.  Implementing  this  suggestion  would  also 
include  printing  the  indices  out  as  part  of  the  ACE 
documentation.  For  large  estimates,  it  would  be  helpful  to 
be  able  to  merge  files.  For  example,  one  estimator  might  be 
working  on  WBS  items  1-25  while  another  estimator  works  on 
26-50.  It  would  be  nice  to  be  able  to  merge  their  files  for 
a  final  product.  In  addition  to  using  a  more  conventional 
notation  for  undefined  values,  suggested  STATPAK 
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enhancements  include  an  option  to  calculate  a  correlation 
matrix  on  the  variables  in  a  data  set  as  well  as  a 
connection  to  a  graphics  capability.  The  graphics  interface 
could  be  resident  within  ACEIT  or  simply  be  a  capability  to 
format  an  output  file  to  be  used  with  an  existing  offline 
graphics  package  such  as  Harvard  Graphics. 

The  most  asked  for  improvement  to  the  User's  Guide  was 
an  index.  Although  the  chapter  table  of  contents  are  quite 
thorough,  an  index  would  be  very  helpful  for  the  novice 
ACEIT  user.  Another  convenient  reference  feature  might  be  a 
keystroke  template  to  lay  over  the  keyboard.  Additional 
information  should  be  provided  on  the  models  connected  to 
ACE  (for  example  source  references).  Some  are  well 
commented,  others  are  not.  Further  explanation  of  the 
sequential/disjoint  learning  curve  routine  would  also  be 
helpful  (again  including  a  source  reference).  Lastly,  the 
confusing  residual  notation  mentioned  in  the  findings 
section  of  this  thesis  should  be  clarified. 

Further  research  with  ACEIT  should  investigate  the 
technicalities  involved  with  creating  the  organization 
specific  databases.  Several  concerns  surfaced  during  this 
research  as  to  the  benefit  of  ACEIT  without  the 
organization-specific  databases  if  the  cost  of  obtaining  and 
maintaining  those  databases  becomes  prohibitive.  Additional 
research  should  also  attempt  to  replicate  an  estimate  from 
scratch  using  ACEIT  to  determine  if  the  internal  linkages  to 
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CERs ,  models,  and  factors  can  accidentally  hurt  instead  of 
help  an  estimator.  For  instance,  one  could  compare  the 
results  of  two  equally  qualified  analysts  accomplishing  the 
same  estimate,  one  using  ACEIT  and  one  not  using  ACEIT. 
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Appendix  A:  Sample  SAS  Regression  Output 


DEP  VARIABLE:  A 


ANALYSIS  OF  VARIANCE 


SUM  OF 

MEAN 

SOURCE 

DF 

SQUARES 

SQUARE 

F  VALUE 

PROB>F 

MODEL 

2 

59654620.98 

29827310.49 

6.996 

0.0494 

ERROR 

4 

17054746.45 

4263686.61 

C  TOTAL 

6 

76709367.43 

ROOT 

'  MSE 

2064.87 

R- SQUARE 

0.7777 

DEP 

MEAN 

4063.714 

ADJ  R-SQ 

0.6665 

C.V. 

50.81237 

PARAMETER  ESTIMATES 

PARAMETER 

STANDARD 

T  FOR  HO: 

VARIABLE 

DF 

ESTIMATE 

ERROR 

PARAMETER=0 

PROB  >  IT! 

INTERCEP 

1  1 

349.33440483 

1195.12748 

0.711 

0.5165 

B 

1 

0.07576375 

0.02602999 

2.911 

0.0437 

C 

1 

5918.77755 

2242.47918 

2.639 

0.0576 

STANDARDIZED 

VARIABLE 

DF 

ESTIMATE 

INTERCEP 

1 

0 

B 

1 

0.68995102 

C 

1 

0.62565428 

OBS 

A 

RESID 

PREDICT 

STUDENT 

1 

7119 

-1190.3 

8309.26 

-1.5251 

2 

642 

-592.7 

1234.74 

-0.3398 

3 

542 

-631.8 

1173.75 

-0.3640 

4 

7755 

3614.3 

4140.66 

1.9395 

5 

2267 

143.4 

2123.60 

0.0779 

6 

8555 

0.0 

8555.00 

• 

7 

1566 

-1343.0 

2908.97 

-0.7144 
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A  I 
9000  + 


PLOT  OF  A*PREDICT 


LEGEND:  A  =  1  OBS,  B  =  2  OBS,  ETC. 


8000  + 


7000  + 


6000  + 


5000  + 


4000  + 


3000  + 


2000  + 


1000  + 


A 


A 


A 


I  AA 

I 

I 

I 

I 

0  + 

1000 


— +_ 
2000 


__+ - + - + - +_ 

3000  4000  5000  6000 

PREDICTED  VALUE 


— +- 
7000 


A 


A 


— +- 
8000 
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A 

9000  + 


8000  + 


7000  + 


6000  + 


5000  + 


4000  + 


3000  + 


2000  + 


1000  + 


PLOT  OF  A*B  LEGEND:  A  =  1  OBS,  B  =  2  OBS,  ETC 


AA 


0  + 
-+- 
0 


20000 


40000 


60000 

B 


80000  100000 


120000 
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Appendix  B;  Sample  STATPAK  Regression  Output 


*****  Linear  Model  ***** 

*****  with  Intercept  *****  i 


THE  REGRESSION  EQUATION  IS: 
A=849. 334+0, 076*8+5918.777*0 


VARIABLE 

COEFFICIENT 

STD  DEV 
OF  COEF 

T-STATISTIC 
(COEF/S  D) 

COEF  NOT  0 
(95%  PROB) 

INTERCEPT 

849.3342 

1195.1274 

0.71 

2.776 

B 

0.0758 

0.0260 

2.91 

2.776 

C 

5918.7773 

2242.4790 

2.64 

2.776 

STD  ERROR  (SE)  =  SQRT(MSE)  =  2064,9 

R-SQUARE  =  77.77  % 

R-SQUARE  =  66.65  %,  ADJUSTED  FOR  DEGREES  OF  FREEDOM  (DF) 


ANALYSIS  OF  VARIANCE  (IN  FIT  SPACE) 


DUE  TO 

DF 

SUM  OF 
SQUARES(SS) 

MEAN  SQ  = 
SS/DF 

F-STAT 

COEF 

(95% 

NOT  0 
PROB) 

REGRESSION 

2 

59654624.000 

29827312.000 

7.00 

6.94 

RESIDUAL (ERROR) 

4 

17054746.000 

4263686.500 

TOTAL 

6 

76709368.000 

FURTHER  ANALYSIS  OF 

VARIANCE  (IN  FIT  SPACE) 

SS  EXPLAINED  BY 

EACH  VARIABLE  WHEN 

ENTERED  IN  THE 

ORDER 

SIVEN 

1  DUE  TO 

DF 

SEQUENTIAL  SS 

1 

1 

1 

! REGRESSION 

2 

1 

59654624.0001 

IB 

1 

29952156.000 

1 

1 

!C 

1 

29702466.0001 
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lOBS 

B 

Y  VALUES 

A 

PREDICTED 

Y  VALUE 

STD  DEV 
PRED  Y 

RESIDUAL 

STD  RES! 

.  .  1 

!  1 

98463.00 

7119.00 

8309.262 

1911.706 

-1190.262 

-1.531 

1  2 

5087.00 

642.00 

1234.744 

1105.217 

-592.744 

-0.341 

1  3 

4282.00 

542.00 

1173.755 

1118.883 

-631.755 

-0.361 

1  4 

43442.00 

7755.00 

4140.664 

889.411 

3614.336 

1.94i 

I  5 

16819.00 

2267.00 

2123.605 

937.128 

143.395 

0.081 

!  6 

23585.00 

8555.00 

8555.000 

2064.870 

0.000 

10000. OOIR 

!  7 

27185.00 

1566.00 

2908.972 

854.453 

-1342.972 

-0.71i 

SE  = 

2064.870, 

MEAN  =  4063 

.71,  COEF  OF  VAR  =  50. 

81%  IN  UNIT 

SPACE 

R  DENOTES  AN  OBSV  WITH  A  LARGE  STD  RESIDUAL 
DURBIN-WATSON  STATISTIC  =  1.89 


Predicted  A 
8560.0  + 


6555.0  + 


4550.0  + 


2545.0  + 


* 


540.0  I- 
540.0 


2545.0  4550.0  6555.0 

Actual  A 

Actual  vs.  Predicted 


8560.0 


61 


Std  Residuals 
10010.0  + 


7505.0  + 


5000.0  + 


2495.0  + 


-10.0 
1170.0 


*  * 

2650.0 


- + - +— 

4130.0  5610.0 

Predicted  A 
Std  Residual  Plot 


7090.0 


•k 

- + 

8570.0 


***********  Percentage  Error  Table  *********** 


Observations 

Actuals 

Predicted 

Residuals 

%  Errors! 

1 

7119.00 

8309.26 

-1190.26 

16.721 

2 

642.00 

1234.74 

-592.74 

92.331 

3 

542.00 

1173.75 

-631.75 

116.561 

4 

7755.00 

4140.66 

3614.34 

-46.61! 

5 

2267.00 

2123.60 

143.40 

-6.33!. 

6 

8555.00 

8555.00 

0.00 

0,00! 

7 

1566.00 

2908.97 

-1342.97 

85.76! 

Avg  (Arith) 

4063.71 

4063.71 

0.00 

36.92% 

Avg  (Absolute) 

1073.64 

52.04% 

Average  Actual  (Avg  Act)  in  Unit  Space .  4063.71 

Standard  Error  (SE)  in  Unit  Space .  2064.87 

Root  Mean  Square  (RMS)  of  %  Errors .  67.6% 

Mean  Absolute  Deviation  (MAD)  of  %  Errors .  52.0% 

Coef  of  Variation  based  on  Std  Error(SE/Avg  Act) .  50.8% 

Coef  of  Variation  based  on  HAD  Res  (MAD  Res/Avg  Act)..  26.4% 
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Appendix  C:  Sample  Hutchison  Program  Learning  Curve  Output 


File  Name  =  hmwkl 
Total  number  of  lots  =  7 


Lot 

First  Unit 

Last  Unit 

Total  Lot 

1 

1 

8 

2312.00 

9 

24 

2672.00 

3 

25 

50 

3120.00 

4 

51 

32 

3040.00 

5 

33 

122 

3000.00 

6 

123 

172 

3500.00 

7 

173 

232 

3660.00 

File  name  =  hmwkl 

Total  number  of  lots  =  7 


Unit  curve  -  ordinary  least  sauares 


First  unit  value 
Slope  coefficient 
Learning  curve  slope 
Coefficient  of  correlation 
Coefficient  of  determination 


473.45 

-.386817 

76.43 

-0.998788 

0.997573 


File  Name  =  hmwkl 
Total  number  of  lots  =  7 


Lot 

Actual  Total 

Cost  of  the  lot 

Predicted  Total 
Cost  of  the  lot 

Difference 

Percentage 

Difference 

1 

2312.00 

2347.64 

-35.64 

-1.5 

2 

2672.00 

2620.67 

51.33 

2.0 

3 

3120.00 

3063.77 

56.23 

1.8 

4 

3040.00 

3003.42 

36.58 

1.2 

5 

3000.00 

3169.91 

-169.91 

-5.4 

6 

3500.00 

3439.08 

60.92 

1.8 

7 

3660.00 

3648.52 

11.48 

0.3 
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H - ( - ( - 1 - 1 - ( - 1 - 1 - 

.04  .323  .606  .339  1.17  1.45  1.73  2.02 

LEARNING  CUR^/E  -  ARITOIEnC  PLOT  -  COSTS  =  1  *  100  UNITS  =  1  *  100 


2.3-  * 

I 

2.3- 

I 

1.9- 

I 

1.6- 

I 

1.3- 
[ 

1.0- 

1 

.9  - 


.74-  * 

I  * 

.61-  * 

^ - , - , - ( - 1 - 1 - 1 - 1 — 

.04  .07  .122  .214  .376  .658  1.15  2.02 

LEARNING  CURVE  -  LOG  LOG  PLOT  -  COSTS  =  1  *  100  UNITS  =  1  *  100 


64 


Appendix  D:  Sample  STATPAK  Learning  Curve  Output 


*****Learning  Curve  Unit  Theory***** 


THE  REGRESSION  EQUATION  IS: 

UNIT  COST=473.324*(CUM  QTV-0.387) 
(T1  =  473.324,  Slope  =  76.48%) 


STD  DEV 

T-STATISTIC 

COEF 

NOT  0 

VARIABLE 

COEFFICIENT 

OF  COEF 

(COEF/S  D) 

(95% 

PROB) 

INTERCEPT 

6.1598 

0.0344 

179.04 

2.015 

CUM  QTY 

-0.3868 

0.0085 

-45.34 

2.015 

Note:  A  5%  cut-off  point  is  listed  for  using  one-sided 
test  in  learning  curve  applications. 


STD  ERROR  (SE)  =  SQRT(MSE)  =  0.0298 
R-SQUARE  =  99.76  % 

R-SQUARE  *  99.71  %,  ADJUSTED  FOR  DEGREES  OF  FREEDOM  (DF) 
ANALYSIS  OF  VARIANCE  (IN  FIT  SPACE) 


DUc  TO 

DF 

SUM  OF 

SQUARES (SS) 

MEAN  SQ  = 

SS/DF 

F-STAT 

COEF 

(95% 

NOT  0 

PROB) 

REGRESSION 

1 

1.829 

1.829 

2055.58 

6.61 

RESIDUAL (ERROR) 

5 

0.004 

0.001 

TOT.^L 

6 

1.834 

SE  ^  0.030,  MEAN  =  4.69,  COEF  OF  VAR  =  0.64%  IN  LOG  SPACE 
Warning;  Observation (s)  with  large  std  residual  (s) 
found  in  your  data  set,  Checlt  input  data!! 
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Predicted  LOT  AVG  HRS 

8.0  + 


j  * 

I  2 

i  3 

4.0  + 


2.0  + 


0.0  2.0  4.0  6.0  8.0 

Actual  LOT  AVG  HRS 
Actual  vs.  Predicted 


***********  Percentage  Error  Table  *********** 


{Observations  Actuals  Predicted  Residuals  %  Errors 

j  _ _ _ _ _ 

Hot  1  289.00  293.48  -4.48  1.55 

Hot  2  167.00  163.78  3.22  -1.93 

Hot  3  120.00  117.83  2.17  -1.81 

Hot  4  95.00  93.86  1.14  -1.20 

Hot  5  75.00  79.25  -4.25  5.66 

Hot  6  70.00  68.78  1.22  -1.74 

{lot  7  61.00  60.81  0.19  -0.31 


Avg  (Arith)  125.29  125.40  -0.11  0.03% 

Avg  (Absolute)  2.38  2.03% 

Average  Actual  (Avg  Act)  in  Unit  Space .  125.29 

Standard  Error  (SE)  in  Unit  Space .  3.35 

Root  Mean  Square  (RMS)  of  %  Errors .  2.6% 

Mean  Absolute  Deviation  (MAO)  of  %  Errors .  2.0% 

Coef  of  Variation  based  on  Std  Error(SE/Avg  Act) .  2.7% 

Coef  of  Variation  based  on  MAD  Res  (MAD  Res/Avg  Act)..  1.9% 
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Appendix  E;  ACEIT  Questionnaire 

1.  Prior  to  this  lesson  did  you  have  any  experience  using  ACEIT?  If 
so,  how  much? 


2.  What  is  your  cost  estimating  experience  other  than  this  AFIT 
Graduate  Program? 

None _ Some  _  Moderate  _  Extensive  _ 

3.  What  is  your  level  of  computer  expertise? 

None  _  Some  _  Moderate  _  Extensive  _ 

4.  Did  you  feel  this  introduction  to  ACEIT  was 

Too  Brief  _  About  Right  _  Too  Long  _ 

5.  Would  you  have  felt  comfortable  doing  the  tutorial  exercise  on 
your  own  (without  classroom  assistance)? 

Yes _ No  _ 

6.  Is  ACEIT's  external  documentation  (checlc  all  that  apply) 

Readable  _  Helpful  _  Adequate  _  Excellent  _ 

Difficult  _ _  A  Waste  _  Well  organized  _ 

Thorough  _  Not  broad  enough  _  Confusing  _ 

7.  Is  ACEIT's  internal  documentation  (checlc  all  that  apply) 

Helpful  _  Confusing  _  Thorough  _ 

Adequate  _  Inadequate  _  Excellent  _ 

8.  How  user-friendly  do  you  feel  ACEIT  is? 

Not  at  all  _ _ Somewhat  _  Very  _ 

9.  Will  this  brief  intro  help  at  your  next  assignment? 

Yes _  No  _  Don’t  Know  _ 

10.  What  would  you  like  to  have  added/deleted  from  this  introduction? 


11.  What  features  do  you  think  should  be  added  to  ACEIT? 


12.  Other  Comments 
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PERIOD  1: 


Briefing  with  slides  which  covers  ACEIT  background,  each  of  the 
various  ACEIT  modules,  and  an  introduction  to  the  Tutorial  (including 
the  configuration  of  the  system  at  AFIT  and  assigning  user  codes  to 
provide  access  to  the  system) . 

Handouts  include:  ACEIT  User's  Guide  and  a  printout  of  the 
Tutorial  documentation  as  an  example. 

PERIOD  2: 

Hands-on  experience.  Walk  the  students  through  the  Tutorial  (with 
each  student  on  their  own  terminal).  Point  out  some  features  of  the 
system  not  illustrated  by  the  tutorial  exercise  (importance  of  indenture 
in  building  the  WBS,  defining  a  WBS  item,  commenting  on  the  notepad, 
examining  the  definition  provided  for  a  CER  in  the  ACE  CER  library) . 
Highlight  things  which  might  be  different  in  the  WHIZBANG  exercise. 

Handouts  include:  ACEIT  Tutorial  Notes  on  how  to  access  the  system 
and  receive  a  printout  with  the  current  AFIT  configuration,  the  WHIZBANG 
exerci-e  for  the  following  periods,  and  the  ACEIT  Questionnaire. 

PERIOD  3: 

Hands-on  experience.  Assist  students  in  completing  the  attached 
WHIZBANG  exercise.  Work  up  to  the  first  ACDB  search  in  the  ACE 
methodology  screen. 

PERIOD  4: 

Hands-on  experience.  Continue  assisting  students  in  completing 
the  WHIZBANG  exercise.  Work  through  obtaining  a  printout  of  the 
complete  documentation.  Collect  the  questionnaires. 


68 


WRIZBANG  EXERCISE 


The  WRIZBANG  is  a  ground  radar  system  being  procured  for  the  Air 
Force.  Only  Full  Scale  Development  is  being  considered.  The  plan  is  to 
buy  two  prototypes. 

The  Prime  Mission  Product  includes: 

A.  Radar  Rardware 

B.  Operational  Software 

CSCI  #1  -  Tracking 
CSCI  #2  -  Display 

DO  NOT  include: 

Operational  site  Activation; 

Common  Support  Equipment; 

Industrial  Facilities; 

Warranty; 

Training; 

or  Software  Engineering/Project  Mgmt  (already  included 
in  COCOMO) 


OTHER  INFORMATION 

The  estimate  should  be  developed  in  By$89  ($M). 
Use  15%  as  the  default  G&A. 

Use  12%  as  the  default  fee. 

Development  planned  for  1  Feb  89  -  30  Sep  91. 


METHODOLOGY 


For  the  Radar  Hardware: 

Use  the  BBEST  System  Development  Cost  Model.  Watch  the  units. 
The  inputs  are: 

UClOO  Production  cost:  1107  ($86  in  K) 
contractor  experience:  high 
system  definition:  good 
NOT  a  joint  service  effort 
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For  the  Operational  Software: 

Use  COCOMO.  Assume  all  new  code. 

The  inputs  are; 

Average  rate/manmonth  is  $10000  ($87) 
for  Tracking: 

19500  DSI 
embedded 

all  nominal  except  - 

RELY  -  very  high 
Database  Size  -  high 
Complexity  -  high 
Execution  Time  -  very  high 
Schedule  -  low 

for  Display: 

14700  DSI 
embedded 

all  nominal  except  - 

RELY  -  very  high 
Database  Size  -  high 
Complexity  -  high 
Execution  Time  -  high 
Schedule  -  low 

For  Software  Integration  &  Checkout  -  20%  of  all  CSCI  costs.  DO 
NOT  include  dollar  units  or  year  as  this  will  be  calculated  in  Base  Year 
dollars. 

Use  Ranter  Factors  for  PMP  I&A  and  Peculiar  Support  Equipment. 

A  throughput  from  the  Program  Office  tells  us  that  SPO  Support 
will  be  $4.7M  ($89)  . 

An  engineering  assessment  for  Engineering  Change  Orders  (ECOs) 
indicates  using  10%  of  the  FSED  WHIZBANG  system  cost. 

Develop  a  CER  for  System  Test  and  Evaluation  using  ACDB  cost 
information  for  FSED  radar  programs  except  programs  3,  6,  8,  9,  12,  13. 
Search  by  Name,  not  Text.  If  you  choose  a  Log  -  Log  regression  include 
the  following  conversion  factor  (this  will  not  be  required  in  the  SEP  89 
release  of  ACEIT) : 

PMP  =  (PMP_F  *  1000)  /  1.1364 

Manload  System  Engineering/Project  Management.  Use  user  defined 
variables  to  make  sensitivity  analysis  easier.  (Include  these  variables 
in  your  equations,  then  do  a  methodology  check  and  let  ACEIT  add  them  to 
the  bottom  of  the  WBS  where  you  can  then  put  in  numerical  values) .  An 
engineering  assessment  indicates: 

people  -  24 

months  -  32 

rate/person/month  -  $2K  ($89) 


70 


Data  costs  are  analogous  to  the  Berlin  Radar  (Templehof)  program. 


ADJUSTMENTS 

Check  the  Adjustments  workscreen  to  ensure  all  appropriate  G&A  and 
fees  have  been  included,  ie.  Do  BBEST  and  COCOMO  include  G&A  and  fees? 
Does  Software  Integration  &  Checkout  need  G&A  and  fees  added  or  are  they 
already  included? 


PHASING 

Use  a  65/45  (65%  spent  at  45%  complete)  Beta  curve  at  the  total 
program  line.  Medium  peakness  is  appropriate  and  the  start  and  end 
dates  are  the  same  as  the  FSED  dates.  The  only  exception  to  this  is 
that  ECO's  and  SPO  support  should  be  phased  using  percent  obligated  as 
follows: 


89 

90 

91 

ECOs 

10 

50 

40 

SPO  Support 

25 

40 

35 
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SHORTENED  WHIZBANG  EXERCISE 


The  WHIZBANG  is  a  ground  radar  system  being  procured  for  the  Air 
Force.  Only  Full  Scale  Development  is  being  considered.  The  plan  is  to 
buy  two  prototypes. 

The  Prime  Mission  Product  includes: 

A.  Radar  Hardware 

B.  Operational  Software 

CSCI  #1  -  Tracking  (expand  to  get  this) 

Software  Integr/Checkout 

DO  NOT  include: 

Operational  site  Activation; 

Common  Support  Equipment; 

Industrial  Facilities; 

Warranty; 

Training; 

or  Software  Engineering/Project  Mgmt  (already  included 
in  COCOMO) 


OTHER  INFORMATION 

The  estimate  should  be  developed  in  BY$89  ($M) . 
Use  15%  as  the  default  G&A. 

Use  12%  as  the  default  fee. 

Development  planned  for  1  Feb  89  -  30  Sep  91. 


METHODOLOGY 


For  the  Radar  Hardware: 

Use  the  BBEST  System  Development  Cost  Model.  Watch  the  units. 
The  inputs  are: 

UClOO  Production  cost:  1107  ($86  in  K) 
contractor  experience:  high 
system  definition:  good 
NOT  a  joint  service  effort 
Note  above  -  2  prototypes. 
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For  the  Operational  Software: 


Use  COCOMO.  Assume  all  new  code. 

The  inputs  are: 

Average  rate/manmonth  is  $10000  ($87) 
for  Tracking: 

19500  DSI 

embedded 

all  nominal  except  - 

RELY  -  very  high 
Database  Size  -  high 
Complexity  -  high 
Execution  Time  -  very  high 
Schedule  -  low 

For  Software  Integration  &  Checkout  -  20%  of  CSCI  costs  (you 
need  to  give  a  variable  name  to  CSCI  and  then  create  an  equation  in  the 
methodology  portion  of  the  software  integration  and  checkout  line  item) . 
DO  NOT  include  dollar  units  or  year  as  they  are  already  associated  with 
the  variable  you  are  using. 

Use  Ranter  Factors  for  PMP  I&A  and  Peculiar  Support  Equipment  (NOT 
the  Ranter  Model) . 

A  throughput  from  the  Program  Office  tells  us  that  SPO  Support 
will  be  $4.7M  ($89) . 

An  engineering  assessment  for  Engineering  Change  Orders  (ECOs) 
indicates  using  10%  of  the  FSED  WHIZBANG  system  cost  (again  assign  a 
variable  name  and  create  an  equation) . 

Develop  a  CER  for  System  Test  and  Evaluation.  Use  ACDB  to 
retrieve  cost  information  for  FSED  radar  programs  (search  by 
program/project/task  name  for  radar  and  only  search  FSED  contracts) . 
Retrieve  all  except  programs  3,  6,  8,  9,  12,  13.  Transfer  the  data  to 
STATPAR  to  do  regressions.  If  you  choose  a  Log  -  Log  regression  include 
the  following  conversion  factor  (this  will  not  be  required  in  the  SEP  89 
release  of  ACEIT) : 

PMP  =  (PMP  F  *  1000)  /  1.1364 


System  Enqineerinq/Project  Management.  Create  an  equation  to 
project  cost  (manpower  x  time  x  rate) .  Use  user  defined  variables  to 
make  sensitivity  analysis  easier.  (Include  these  variables  in  your 
equations,  then  do  a  methodology  check  and  let  ACEIT  add  them  to  the 
bottom  of  the  WBS  where  you  can  then  put  in  numerical  values) .  An 
engineering  assessment  indicates: 
people  -  24 
months  -  32 

rate/person/month  -  $2R  ($39) 
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Data  costs  are  analogous  to  the  Berlin  Radar  (Templehof)  program. 
Search  ACDB  by  program/project/task  name,  move  the  data  to  STATPAK  and 
retrieve  only  the  data  costs. 


ADJUSTMENTS 

Check  the  Adjustments  workscreen  to  ensure  all  appropriate  GiA  and 
fees  have  been  included,  ie.  Do  BBEST  and  COCOMO  include  G&A  and  fees? 
Does  Software  Integration  &  Checkout  need  G&A  and  fees  added  or  are  they 
already  included? 


PHASING 

Use  Beta  curve  phasing  for  the  total  system  and  use  percent 
obligated  for  ECOs  and  SPO  Support. 

Use  a  65/45  (65%  spent  at  45%  complete)  Beta  curve  at  the  total 
program  line.  Medium  peak;  5s  is  appropriate  and  the  start  and  end 
dates  are  the  same  as  the  FSED  dates.  ECO's  and  SPO  support  should 
be  phased  using  percent  obligated  as  follows; 


89 

90 

91 

ECOs 

10 

50 

40 

SPO  Support 

25 

40 

35 

That's  all  .... 

Congratulations!  You  have  finished  your  first  ACEIT  estimate.  Now, 
print  out  a  complete  copy  of  all  the  documentation  and  relax  -  -  -  until 
someone  asks  for  a  what-if  analysis! 


Extracted  From  ACEIT  STATPAK  Practice  Data  Set  (13:Sec  9,8) 


System 

A 

B 

C 

1 

7119 

98463 

0 

2 

642 

5087 

0 

3 

542 

4282 

0 

4 

7755 

43442 

0 

5 

2267 

16819 

0 

6 

8555 

23585 

1 

7 

1566 

27185 

0 

Note:  C  is 

a  dummy  variable 

where  yes=l,  No=0, 
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Helicopter  Turboshaft  Engine  Data  Set  (16) 


Engine 

Model 

Cost 

Air 

Flow 

Pressure 

Ratio 

Horsepower 

Weight 

T63 

42.28 

3.20 

6.25 

317 

133 

T53 

51.96 

10.45 

5.80 

860 

480 

T55 

136.45 

21.50 

6.60 

2200 

570 

T58 

139.76 

12.60 

8.15 

1250 

295 

T56 

157.91 

32.50 

9.10 

3755 

1833 

T64 

185.37 

25.50 

12.20 

2690 

717 

T700 

193.55 

9.50 

17.10 

1560 

415 

T73 

286.46 

52.20 

6.80 

4050 

713 

Engine  Model 

Horsepower /Weight 

T63 

2.29710 

T53 

1.79167 

T55 

3.85965 

T58 

4.23729 

T56 

2.04855 

T64 

3.75174 

T700 

3.75904 

T73 

5.68022 
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Solid  Propellant  Rocket  Motor  Data  Set  (17) 


Motor 

Cost 

{150th 

unit) 

Propellant 

Weight 

(lb) 

Inert 

Weight 

(lb) 

Total 

Weight 

(lb) 

Burn 

Time 

(sec) 

1 

2097 

6.8 

10.8 

16.9 

1.35 

2 

3542 

5.7 

7.3 

13.0 

1.50 

3 

data  omitted 

from  calculations 

4 

8019 

60.0 

24.0 

84.0 

2.29 

5 

8607 

65.0 

38.0 

103.0 

4.14 

6 

14660 

60.0 

39.0 

99.0 

5.21 

7 

13310 

90.8 

71.5 

162.3 

3.04 

8 

25269 

243.0 

274.0 

517.0 

1.52 

9 

*  data  omitted 

from  calculations 

10 

80796 

361.0 

78.0 

439.0 

27.40 

11 

55520 

605.0 

245.0 

850.0 

31.50 

12 

47805 

562.0 

228.0 

790.0 

22.40 

13 

46452 

725.0 

416.0 

1141.0 

18.00 

14 

“  data  omitted 

from  calculations 

15 

94594 

2785.0 

463.0 

3248.0 

39.00 

16 

556548 

3665.0 

640.0 

4305.0 

56.00 

17 

380124 

3657. o' 

592.0 

4249.0 

56.40 

18 

90743 

4451.0 

553.0 

5004.0 

38.30 

19 

169916 

5900.0 

1010.0 

6910.0 

23.80 

20 

198930 

6562.0 

1257.0 

7819.0 

32.20 

21 

*  data  omitted 

from  calculations 

22 

418033 

7301.0 

740.0 

8047.0 

59.60 

23 

687441 

8872.0 

617.0 

9489.0 

75.80 

24 

1125507 

10371.0 

1028.0 

11399.0 

60.10 

25 

636064 

10249.0 

1408.0 

11657.0 

59.30 

26 

780611 

19270.0 

2892.0 

22162.0 

50.60 

27 

1 

»  data  omitted 

from  calculations 

28 

1359457 

20778.0 

3133.0 

23911.0 

59.50 

29 

804248 

38813.0 

3040.0 

41853.0 

66.00 

30 

1235237 

45350.0 

4650.0 

50000.0 

59.40 

*  data  omitted  to  validate  predictive  accuracy  of  model 
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Learning  Curve  Unit  Data  Sets  (11) 


Set  1 


Lot 

Lot  Averaae  Hours 

Cumulative  Ouantitv 

1 

289 

i 

2 

167 

24 

3 

120 

50 

4 

95 

82 

5 

75 

122 

6 

70 

172 

7 

61 

232 

Lot 

Set  2 

Lot  Averaae  Cost 

Cumulative  Ouantitv 

1 

1068.75 

16 

2 

745.00 

36 

3 

630.00 

61 

4 

545.16 

86 

5 

493.63 

121 

6 

463.77 

1560 
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Longley  Data  Set  (14:832) 


YEAR 

XI 

X2 

X3 

X4 

X5 

X6 

y 

1947 

83.0 

234289 

2356 

1590 

107608 

1947 

60323 

1948 

88.5 

259426 

2325 

1456 

108632 

1948 

61122 

1949 

88.2 

258054 

3682 

1616 

109773 

1949 

60171 

1950 

89.5 

284599 

3351 

1650 

110929 

1950 

61187 

1951 

96.2 

328975 

2099 

3099 

112075 

1951 

63221 

1952 

98.1 

346999 

1932 

3594 

113270 

1952 

63639 

1953 

99.0 

365385 

1870 

3547 

115094 

1953 

64989 

1954 

100.0 

363112 

3578 

3350 

116219 

1954 

63761 

1955 

101.2 

397469 

2904 

3048 

117388 

1955 

66019 

1956 

104.6 

419180 

2822 

2857 

118734 

1956 

67857 

1957 

108.4 

442769 

2936 

2798 

120445 

1957 

68169 

1958 

110.8 

444546 

4681 

2637 

121950 

1958 

66513 

1959 

112.6 

482704 

3813 

2552 

123366 

1959 

68655 

1960 

114.2 

502601 

3931 

2514 

125368 

1960 

69564 

1961 

115.7 

518173 

4806 

2572 

127852 

1961 

69331 

1962 

116.9 

554894 

4007 

2827 

130081 

1962 

70551 

79 


Bibliography 


1.  AFIT  GCA  88S.  "Preliminary  Evaluation  of  the  Automated 
Cost  Estimating  Integrated  Tools  (ACEIT)  System.." 
^Unpublished  study.  Air  Force  Institute  of  Technology, 
Wriaht-Patterson  AFB  OH,  1988. 

2.  .Albee  Cliff.  Operations  Research  Analyst.  Telephone 
interview.  BSD/ACC,  Norton  AFB  CA.  22  May  1989. 

3.  Birkofer,  Kenneth  L.  A  Review  of  the  ESP  .ACE-IT 
(.Autom.atic  Cost  Estim.ator)  Model.  Unpublished  report. 
ASD/ACCR",  Wright-Patterson  AFB  OH,  February  1989. 

4.  Bradley,  Randall  S.  Evaluation  of  the  Automated  Cost 

mating  Integrated  Tools  (ACEIT)  System .  Research 
Project  presented  to  ASD/ACC  in  partial  fulfill.ment  cf 
the  requirements  for  the  journeyperson  level  in 
Financial  Management.  ASD/ACC,  Wright-Patterson  AFB 
OH.  May  1989. 

5.  Coaklev,  Ellen,  Technical  Director  of  Cost  .Analysis. 

Personal  interviews.  ESD/.ACC,  Hanscom.  .AFB  M.A .  4-5  ‘fay 

1989  . 

6.  Davis,  Gordon  B.  and  Margrethe  H,  Olson.  Man agemen t 
Information  Sys  tem.s  f  Second  Edition)  .  r^ew  York: 
McGraw-Hill  Book  Company,  1985. 

.  Dean,  Captain  Joe,  Senior  Software  Cost  Analyst. 

Telephone  interview.  ESD/ACC,  Hanscom  .AFB  M.A,  12  July 
1939  . 

3.  Delta  Research  Corporation.  Cost  Estimating  System. 
Volum.e  1,  Autom.ated  Cost  Estimating  System ,  November 
1987.  Contract  F08635-35-C-0230 .  .Arlington  7A . 

9.  Kanson,  David,  Director  of  Cost.  Telephone  interviov. 
BSD/ ACC,  Norton  AFB  CA ,  31  July  1989. 

10.  Hutchison,  Capt  Larry  D.  .A  Hicroco.mputer  Progra.m  for 
the  Solution  of  Learning  Curve  Com.putations  .  MS 
thesis,  AFIt7gsm7lSG/S5S-13 .  School  of  Systems  and 
Logistics,  Air  Force  Institute  of  Technology  (.AU) 
Wright-Patterson  AFB  OH,  September  1985. 

11.  Kankev,  Roland  D.  Class  handout  distributed  in  COST 
673,  Cost  Estimating  for  Weapon  System  Production. 
School  of  Systems  and  Logistics,  Air  Force  Institute  of 
Technology  (AU) ,  Wright-Patterson  AFB  OH,  May  1989. 


30 


12.  Kazanowski,  A.  D.  "Cost-Effectiveness  Fallacies  and 
Misconceptions  Revisited,"  Cost-Effectiveness;  The 
Economic  Evaluation  of  Engineered  Systems,  edited  by  J. 
Morely  English.  New  York:  John  Wiley  &  Sons,  Inc., 
1968. 

13.  Kielpinski,  T.  J.  and  others.  CR-0300/2  Automated  Cost 
Estimating  Integrated  Tools  (ACEIT)  User's  Guide. 
Version  2.5,  January  1989.  Contract  F19628-88-D-0007 . 
Santa  Barbara  CA;  Tecolote  Research,  Inc. 

14.  Longley,  James  W.  "An  Appraisal  of  Least  Squares 
Programs  for  the  Electronic  Computer  from  the  Point  of 
View  of  the  User,"  Journal  of  the  American  Statistical 
Association.  62:  819-841  (1967). 

15.  Murphy,  Richard  L.  Assistant  Professor.  Personal 
interviews.  AFIT/LSQ,  Wright-Patterson  AFB  OH, 

January  -  August  1989. 

16.  Murphy,  Richard  L.  Class  handout  distributed  in  COST 

671,  Cost  Modeling.  School  of  Systems  and  Logistics, 
Air  Force  Institute  of  Technology  (AU) ,  Wright- 
Patterson  AFB  OH,  December  1988. 

17.  Murphy,  Richard  L.  Class  handout  distributed  in  COST 

672,  Model  Diagnostics  and  Software  Management.  School 
of  Systems  and  Logistics,  Air  Force  Institute  of 
Technology  (AU) ,  Wright-Patterson  AFB  OH,  March  1989. 

18.  Netter,  John  and  others.  Applied  Linear  Regression 
Models .  Homewood  IL:  Irwin,  1983. 

19.  Raduchel,  William  J.  "A  Professional's  Perspective  on 
User-Friendliness,"  Byte,  9:  101-106  (May  1984). 

20.  Shell,  Beau.  "Thinking  About  Artificial  Intelligence," 
Harvard  Business  Review.  65:  91-97  (July-August  1987). 

21.  Simpscn,  Henry.  Design  of  User-Friendly  Programs  for 
Small  Computers.  New  York:  McGraw-Hill,  1985. 

22.  Voss,  Captain  Chis,  Cost  Analyst.  Telephone 
interview.  SSD/ACC,  Los  Angeles  AFB  CA,  12  July  1989. 

23.  Weech,  Margaret,  ACEIT  Program  Manager.  Personal  and 
telephone  interviews.  ESD/ACC,  Hanscom  AFB  MA, 

May  -  July  1989. 


81 


24.  Williams,  Dorothy  D.  and  Roger  A.  Dietrich.  "Beyond 
the  DCA-P-92(r)  Structure  Costing  at  a  New  Level  of 
Detail."  US  Army  Communications-Electronics  Command, 
Fort  Monmouth  NJ. 


82 


UNCLASSIFIED 


SECURITY  CLASSIFICATION  OF  THIS  PAGE 


la,  REPORT  SECURITY  CLASSIFICATION 

UNCLASSIFIED 


2a.  SECURITY  CLASSIFICATION  AUTHORITY 


2b.  DECLASSIFICATION /DOWNGRADING  SCHEDULE 


4.  PERFORMING  ORGANIZATION  REPORT  NUMBER(S) 

AFIT/GCA/LSQ/8  9S- 5 


REPORT  DOCUMENTATION  PAGE 


lb.  RESTRICTIVE  MARKINGS 


Form  Approved 
0MB  No.  0704-0188 


3.  DISTRIBUTION /AVAILABILITY  OF  REPORT 

Anproved  for  public  release; 
distribution  unlimited 


5.  MONITORING  ORGANIZATION  REPORT  NUMBER(S) 


6a.  NAME  OF  PERFORMING  ORGANIZATION  6b.  OFFICE  SYMBOL 
School  of  Systems  (if  applicable) 

and  Logistics  AFIT/LSQ 


6c  ADDRESS  (Oty,  State,  and  ZIP  Code) 

Air  Force  Institute  of  Technologv 
Uright-Patterson  AFB  OH  45433-6583 


6b.  OFFICE  SYMBOL  I  7a.  NAME  OF  MONITORING  ORGANIZATION 
(If  applicable)  I 


8a.  NAME  OF  FUNDING /SPONSORING 
ORGANIZATION 


8c  ADDRESS  (City,  State,  and  ZIP  Code) 


8b.  OFFICE  SYMBOL 
(If  applicable) 


7b.  ADDRESS  {City,  State,  and  ZIP  Code) 


9  PROCUREMENT  INSTRUMENT  IDENTIFICATION  NUMBER 


10.  SOURCE  OF  FUNDING  NUMBERS 


PROGRAM 

PROJECT 

TASK 

ELEMENT  NO. 

NO. 

NO 

WORK  UNIT 
ACCES5  0N  NO. 


11.  TITLE  (Include  Securiy  Classification) 

AN  EVALUATION  OF  THE  AUTOMATED  COST  ES'^I'UL'^ING 
TOOLS  (ACEIT)  SYSTEM 


12.  PERSONAL  AUTHO.R{S) 

Caroline  L.  Hanson,  3. A.,  M.3.A.,  .Major,  USA? 


13a.  TYPE  OF  REPORT  I 

.MS  Thesis  I 

13b.  TIME  COVERED 

FROM  TO 

14.  DATE  OF  REPORT  (Year,  Month,  Day) 

1939  September 

:5  ?AG= 

94  ‘ 

18.  SUBiECT  TERMS  (Continue  on  reverse  if  necessary  and  identify  by  block  nu  —  oer) 


Cost  Analysis 
Cost  Estimates 


■9.  ABSTRACT  (Conti,-.ue  on  reverse  if  necessary  and  identify  by  block  number) 


Cost  Effectiveness 
Cost  .Models 


Thesis  .Tdvisor:  Poland  D.  Mankev,  ?h.O. 

Associate  Professor 
Decartm^ent  of  Cuanr iraoive  ."am 


Approved  fQr  ,pub1ic  release:  lAU  APR  190-1. 
LAPRy  i.  EK.YELHAINZ.  Lt^Col .  USAF 


LAPRy  i.  EK.YELHAINZ.  Lt/Col .  USAF  11  Oct  39 
Director  of  Research  and  Consultation 
Air  Force  Institute  of  Technology  (AU) 
Wright-Patterson  AFB  OH  <5433-6583 


20.  D'STRlBUTlON/AVAILABILITY  OF  ABSTRACT 

C5uNCLASS;F:£0.'UNLIMiT£0  □  SAME  AS  RPT.  □  OTIC  USERS 

21,  abstract  security  CLASS'FiCAi 
UNCLASSIFIED 

■|ON 

22a  NAME  OP  RESPONSIBLE  INDIVIDUAL 

Roland  D.  Kankey,  Assoc.  Professor 

22b  telephone  (Include  Area  Code) 

(513)  255-8410 

22c.  OFFICE  SYMBOL 

LSQ 

DDForm  1473,  JUN  86 


Previous  editions  are  obsolete. 

/i 


SECURITY  CLASSlFlCAT  ON  QF  --  $  =AGE 

UNCLASSIFIED 


UNCLASSIFIED 


he  purpose  of  this  study  was  to  evaluate  the  Automated 
Cost  Estimating  Integrated  Tools  (ACEIT)  System  and  answer 
the-(^gl-lowing— inves  tigative  quest  ion  s-fL.^  \  r\ 

'V  lVfl_  T- 
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the  system? 


A  literature  review,  discussions  with  ACEJIT 
AFIT  students,  personal  experience  with  the 
data  sets  were  used  to  answer  the  questions, 
wer e  n 

1)  ACEIT  is  very  user-friendly  and  does  not  r 
extensive  training  although  training  should  be 
those  that  desire  or  need  it'. 

52)  Since  ACEIT  is  not  an  expert  ' system ,  some 
estimating  expertise  is  required  to  generate  a 
estimates . 

^3)  Sample  data  sets  run  through  the  ACEIT  Sta 
Package  and  SAS  produced  comparable  estimates 
for  learning  curve,  linear  and  log-linear  rear 
^4)  ACEIT' s  m.a.icr-  benefit,  being  able  to  fill 
with  organisation  specific  work  breakdown  stru 
and  cost  estimating  relationships,  may  also  be 
potential  pitfall.  Each  organization  using  AC 
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ensure  ruality  control  of  its  own  unioue  datab- 
^■)  A  brief  ACEIT  introduction  given  to  AFIT  s 
indicated  the  system  can  be  quickly  and  effect 
introduced  .  'T  .  CK 
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